You now have two pieces: the sigmoid, which turns a score into a probability, and gradient descent, which lowers a loss by walking downhill. Logistic regression is what you get when you bolt them together. The model predicts a probability with the sigmoid, you measure its wrongness with a loss built for probabilities, and gradient descent nudges the weights until that wrongness is as small as it can get. Nothing new in the mechanics — just a loss function chosen so that the downhill direction actually exists and leads somewhere good. By the end of this lesson you will have trained a classifier from scratch and watched it correctly label points it has never seen.
The model: a line, then a sigmoid
Logistic regression keeps the linear score you already know, z = w times x plus b, and feeds it through the sigmoid to get a probability: p = sigmoid(w*x + b). That probability is the model's estimate that the example belongs to the positive class. The only difference from the regression line is the sigmoid on top, which is what lets the output be read as a probability instead of an unbounded number. Learning means finding the w and b that make these probabilities line up with the true labels as well as possible.
import math
def log_loss(probs, ys):
total = 0.0
for p, y in zip(probs, ys):
total += y * math.log(p) + (1 - y) * math.log(1 - p)
return -total / len(probs)
print(round(log_loss([0.9], [1]), 4))
print(round(log_loss([0.5], [1]), 4))
The loss: log-loss, built for probabilities
Mean squared error was the right loss for predicting numbers, but it is the wrong tool here. The natural loss for a probability is log-loss (also called binary cross-entropy): for each example you take the log of the probability you assigned to the correct class, and average the negatives. Confident-and-right costs almost nothing; confident-and-wrong costs a fortune, because the log of a tiny probability plunges toward negative infinity. That steep punishment is the whole point — it shoves the model away from confident mistakes far harder than a squared error would, and it hands gradient descent a clean, convex surface to descend instead of a bumpy one.
flowchart LR A["confident and right<br/>p near 1, y = 1"] --> S["small cost<br/>-log(p) near 0"] B["confident and wrong<br/>p near 0, y = 1"] --> L["huge cost<br/>-log(p) explodes"] C["undecided<br/>p = 0.5"] --> M["moderate cost<br/>0.693"] style S fill:#166534,color:#fff style L fill:#b91c1c,color:#fff style M fill:#b45309,color:#fff
import math
def sigmoid(z):
if z >= 0:
return 1 / (1 + math.exp(-z))
e = math.exp(z)
return e / (1 + e)
xs = [1.0, 2.0, 3.0, 4.0]
ys = [0, 0, 1, 1]
n = len(xs)
gw = sum((sigmoid(0.0 * x + 0.0) - y) * x for x, y in zip(xs, ys)) / n
gb = sum((sigmoid(0.0 * x + 0.0) - y) for x, y in zip(xs, ys)) / n
print('grad w =', round(gw, 4), 'grad b =', round(gb, 4))
The gradient: stunningly simple
Here is the part that makes logistic regression a pleasure to build by hand. The gradient of the mean log-loss with respect to the weight w works out to the mean of (p - y) * x over the data — no logs, no exponentials left, just the prediction error times the input. The bias gradient is the mean of (p - y). That clean form is not a coincidence; it falls out of the maths because the sigmoid and the logarithm were made for each other, their messy parts cancelling exactly. The update rule is the same one you wrote for regression: subtract the learning rate times the gradient, and repeat.
flowchart TD A["start w = 0, b = 0"] --> B["compute p = sigmoid(w*x+b)<br/>for every point"] B --> C["gradient: mean of (p - y) * x<br/>and mean of (p - y)"] C --> D["update w, b against the gradient"] D --> B B --> E["loss stops falling:<br/>classify at the 0.5 threshold"] style C fill:#10b981,color:#fff style E fill:#166534,color:#fff
import math
def sigmoid(z):
if z >= 0:
return 1 / (1 + math.exp(-z))
e = math.exp(z)
return e / (1 + e)
def train(xs, ys, lr=0.5, steps=2000):
w = b = 0.0
n = len(xs)
for _ in range(steps):
gw = sum((sigmoid(w * x + b) - y) * x for x, y in zip(xs, ys)) / n
gb = sum((sigmoid(w * x + b) - y) for x, y in zip(xs, ys)) / n
w -= lr * gw
b -= lr * gb
return w, b
def predict(xs, w, b):
return [1 if sigmoid(w * x + b) >= 0.5 else 0 for x in xs]
train_xs = [1.0, 2.0, 5.0, 6.0]
train_ys = [0, 0, 1, 1]
held_x = [0.0, 10.0]
held_y = [0, 1]
w, b = train(train_xs, train_ys)
print('training predictions:', predict(train_xs, w, b))
print('held-out predictions:', predict(held_x, w, b))
print('held-out true labels:', held_y)
Classifying points it has never seen
Training only counts if the model generalises to new data, so the honest test is to hold out a point, train on the rest, and ask the model to label the one it never saw. To turn a probability into a label you threshold at 0.5: probability at or above half is class 1, below is class 0. When the held-out point lands on the correct side of the learned boundary, the model has learned something real rather than memorising the training set. This train-then-test-on-unseen-data loop is the skeleton of every honest evaluation you will ever run, from this toy up to a production model.
flowchart LR N0["x below the boundary<br/>p below 0.5 means class 0"] --> BD["boundary x = -b / w<br/>p = 0.5"] BD --> N1["x above the boundary<br/>p above 0.5 means class 1"] style N0 fill:#b91c1c,color:#fff style BD fill:#b45309,color:#fff style N1 fill:#166534,color:#fff
Reading the weights you learned
Once training finishes, the weights tell a story you can read. The sign of w shows which direction the feature pushes the probability: positive w means larger x raises the chance of class 1, negative w means it lowers it. The size of w is how strongly the feature matters, and the bias b shifts where the boundary sits along the x-axis. Together they fix the boundary at the single point x = -b / w, where the model is exactly undecided. Inspecting these numbers after training is how you sanity-check a model: a weight with the wrong sign, or a boundary in an impossible place, is a clue that something upstream — the data, the features, or the loss — has gone quietly wrong.
What does this print? Each line is the log-loss of a single example, rounded to four decimals.
import math
def log_loss(probs, ys):
total = 0.0
for p, y in zip(probs, ys):
total += y * math.log(p) + (1 - y) * math.log(1 - p)
return -total / len(probs)
print(round(log_loss([0.9], [1]), 4))
print(round(log_loss([0.5], [1]), 4))
For y = 1, only the y * log(p) term survives.
log(0.9) is about -0.1054, negated and rounded gives 0.1054.
log(0.5) is about -0.6931, negated and rounded gives 0.6931.
Write log_loss(probs, ys) returning the mean binary cross-entropy. Clip each probability into [1e-12, 1 - 1e-12] before taking logs so a 0 or 1 never crashes math.log.
import math
def log_loss(probs, ys):
# mean of -( y*log(p) + (1-y)*log(1-p) ); clip p first
pass
Loop over zip(probs, ys), accumulating y*log(p) + (1-y)*log(1-p).
Return the negative of the mean.
Clip with min(max(p, lo), hi) before logging.
import math
def log_loss(probs, ys):
total = 0.0
for p, y in zip(probs, ys):
p = min(max(p, 1e-12), 1 - 1e-12)
total += y * math.log(p) + (1 - y) * math.log(1 - p)
return -total / len(probs)
Write gradient(xs, ys, w, b) returning a tuple (gw, gb) for the logistic model, where gw is the mean of (p - y) * x and gb is the mean of (p - y), with p = sigmoid(w*x + b). Include a self-contained sigmoid (use the stable form).
import math
def sigmoid(z):
# stable sigmoid: handle very negative z without overflow
pass
def gradient(xs, ys, w, b):
# gw = mean of (p - y) * x ; gb = mean of (p - y)
pass
At w = 0 and b = 0 the sigmoid is exactly 0.5 for every point.
With p = 0.5, each (p - y) is +0.5 for class 0 and -0.5 for class 1.
Average the per-point (p - y) * x for gw, and the per-point (p - y) for gb.
import math
def sigmoid(z):
if z >= 0:
return 1 / (1 + math.exp(-z))
e = math.exp(z)
return e / (1 + e)
def gradient(xs, ys, w, b):
n = len(xs)
gw = sum((sigmoid(w * x + b) - y) * x for x, y in zip(xs, ys)) / n
gb = sum((sigmoid(w * x + b) - y) for x, y in zip(xs, ys)) / n
return gw, gb
Log-loss must be a positive number, but this version returns a negative one because it forgot to negate the accumulated sum of logs. Fix the return so the loss comes out positive.
import math
def log_loss(probs, ys):
total = 0.0
for p, y in zip(probs, ys):
total += y * math.log(p) + (1 - y) * math.log(1 - p)
return total / len(probs)
Logs of probabilities are negative, so their sum is negative.
A loss must be positive, so negate the sum before dividing.
import math
def log_loss(probs, ys):
total = 0.0
for p, y in zip(probs, ys):
total += y * math.log(p) + (1 - y) * math.log(1 - p)
return -total / len(probs)
Put it all together. train(xs, ys, lr=0.5, steps=2000) starts at w = 0, b = 0 and applies the gradient update w -= lr*gw, b -= lr*gb each step, returning (w, b). predict(xs, w, b) returns 1 when sigmoid(w*x + b) >= 0.5 else 0. Use the provided stable sigmoid so weights can grow large without overflowing.
import math
def sigmoid(z):
if z >= 0:
return 1 / (1 + math.exp(-z))
e = math.exp(z)
return e / (1 + e)
def train(xs, ys, lr=0.5, steps=2000):
# start w = b = 0; loop the gradient update; return (w, b)
pass
def predict(xs, w, b):
# 1 if sigmoid(w*x + b) >= 0.5 else 0, for each x
pass
Inside the loop, recompute gw and gb from the current w and b, then update both.
predict thresholds each probability at 0.5 with a list comprehension.
The data is linearly separable, so after enough steps every training point is classified correctly and the held-out points follow the same boundary.
import math
def sigmoid(z):
if z >= 0:
return 1 / (1 + math.exp(-z))
e = math.exp(z)
return e / (1 + e)
def train(xs, ys, lr=0.5, steps=2000):
w = 0.0
b = 0.0
n = len(xs)
for _ in range(steps):
gw = sum((sigmoid(w * x + b) - y) * x for x, y in zip(xs, ys)) / n
gb = sum((sigmoid(w * x + b) - y) for x, y in zip(xs, ys)) / n
w -= lr * gw
b -= lr * gb
return w, b
def predict(xs, w, b):
return [1 if sigmoid(w * x + b) >= 0.5 else 0 for x in xs]
Recap
- Logistic regression predicts a probability as
sigmoid(w*x + b)and classifies by thresholding that probability at 0.5. - Log-loss (binary cross-entropy) is the probability-aware loss: confident-and-wrong is punished steeply, confident-and-right costs almost nothing.
- Its gradient is beautifully simple: the mean of
(p - y) * xfor w, and the mean of(p - y)for b. - The update rule is unchanged from regression: subtract the learning rate times the gradient, then repeat.
- Evaluate honestly by training on some data and testing on a held-out point the model never saw.
- On linearly separable data the weights run away toward overconfidence; regularisation (scikit-learn's
C) is the standard cure.
You can now build a binary classifier from scratch. Next you will study the decision boundary it draws — where it places the line, and the shapes of data it simply cannot separate.