Imagine you are predicting house prices from two numbers: the number of bedrooms (1-5) and the square footage (1,000-3,000). To a distance-based algorithm like k-nearest neighbours, the square footage will dominate completely because its numbers are hundreds of times larger. The bedroom count — which might be the better predictor — is treated as noise. That is the scaling problem: features measured in different units pull algorithms toward the largest magnitudes, not the most informative ones. Gradient descent suffers too. When one weight axis is stretched, the loss surface becomes a narrow valley, so the optimizer bounces back and forth and crawls toward the minimum. Scaling does not change the information in the data; it changes the geometry so the algorithm can use it fairly.
flowchart LR
subgraph Before["Before scaling"]
A1["bedrooms: 2"] --> D1["distance dominated by sqft"]
A2["sqft: 2000"] --> D1
end
subgraph After["After scaling"]
B1["bedrooms: 0.25"] --> D2["both axes contribute equally"]
B2["sqft: 0.50"] --> D2
end
Min-max scaling: pin everything to [0, 1]
Min-max scaling squeezes each feature into the range 0 to 1 by subtracting the minimum and dividing by the full spread. For a single list of numbers, the formula is scaled = (x - min) / (max - min). Every value becomes a fraction of the way from the smallest observation to the largest. A house with the minimum square footage becomes 0, the maximum becomes 1, and everything else lands proportionally in between. The shape of the distribution is preserved — the order of the values does not change — only the ruler changes. This is the right choice when you know the bounds matter and you want to keep them, such as pixel intensities that are naturally capped at 255.
def min_max_scale(data):
lo = min(data)
hi = max(data)
return [(x - lo) / (hi - lo) for x in data]
prices = [10, 20, 30, 40, 50]
scaled = min_max_scale(prices)
print([round(x, 2) for x in scaled])
Look at the output: the smallest price became exactly 0, the largest became 1, and the midpoint became 0.5. Those numbers are no longer dollars; they are positions on a relative ruler. That is the key mental shift — scaling replaces the original unit with a proportion. When you feed these scaled values into k-nearest neighbours, a difference of 0.1 in bedrooms carries the same geometric weight as a difference of 0.1 in price, because both are now measured on the same 0-to-1 ruler. The algorithm no longer cares that one feature used to be priced in thousands and the other counted in rooms.
Standard scaling: centre on zero, measure in standard deviations
Standard scaling — also called z-score normalisation — transforms each value by subtracting the mean and dividing by the standard deviation. The result is a distribution centred at 0 with a spread of 1. Values above the mean become positive, values below become negative, and the magnitude tells you how many standard deviations away from average the point is. This is the default choice for gradient descent because it makes the loss surface roughly spherical, so every weight updates at a similar rate. Unlike min-max, standard scaling has no fixed ceiling, so a single extreme outlier does not squash every other value into a tiny band near zero.
import math
def mean(data):
return sum(data) / len(data)
def std_dev(data):
m = mean(data)
return math.sqrt(sum((x - m) ** 2 for x in data) / len(data))
def standard_scale(data):
m = mean(data)
s = std_dev(data)
return [(x - m) / s for x in data]
heights = [150, 160, 170, 180, 190]
scaled = standard_scale(heights)
print([round(x, 2) for x in scaled])
The first height became roughly negative one, the middle height became zero, and the last became roughly positive one. That is exactly what standard scaling promises: the mean is the new zero, and every other value is expressed in multiples of the standard deviation. A result of -1.22 means that point is 1.22 standard deviations below the average. This unit — the standard deviation — is the natural language of probability and optimisation, which is why most deep-learning frameworks default to some form of standard normalisation.
flowchart TD
A["raw feature"] --> B{"bounded range?"}
B -->|"yes"| C["min-max scaling"]
B -->|"no"| D["standard scaling"]
C --> E["values in [0, 1]"]
D --> F["mean 0, std 1"]
style C fill:#166534,color:#fff
style D fill:#3776ab,color:#fff
Why distance and gradient methods care
k-nearest neighbours decides which points are close by measuring straight-line distance. If one feature spans thousands and another spans ones, the large feature acts as the only compass and the small one is ignored. After scaling, both features share the same unit — standard deviations or fractions of the range — so distance means what you think it means. Gradient descent cares for a different reason. An unscaled loss surface is stretched along the axis of the largest feature, so the gradient points mostly sideways and the optimiser zig-zags down a narrow canyon. Scaling turns that canyon into a gentle bowl, and the same learning rate reaches the bottom in far fewer steps. The improvement is not cosmetic; it can be the difference between convergence and divergence.
flowchart TD A["unscaled loss surface"] --> B["narrow valley along one axis"] B --> C["gradient zig-zags<br/>slow convergence"] D["scaled loss surface"] --> E["roughly circular bowl"] E --> F["gradient points straight<br/>to the minimum"] style A fill:#b45309,color:#fff style D fill:#166534,color:#fff
import math
def distance(a, b):
return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))
# bedrooms, sqft
house_a = [3, 2500]
house_b = [2, 2400]
print("before:", round(distance(house_a, house_b), 2))
bedrooms = [2, 3, 4, 3, 2]
sqft = [2400, 2500, 3000, 2600, 2200]
def min_max_scale(data):
lo = min(data)
hi = max(data)
return [(x - lo) / (hi - lo) for x in data]
s_bed = min_max_scale(bedrooms)
s_sqft = min_max_scale(sqft)
# house_a is index 1, house_b is index 0 in the dataset
a_scaled = [s_bed[1], s_sqft[1]]
b_scaled = [s_bed[0], s_sqft[0]]
print("after:", round(distance(a_scaled, b_scaled), 2))
Compare the two distances. Before scaling, the gap looks enormous — a hundred units — because it is driven almost entirely by square footage. After scaling, the distance shrinks to about half a unit, and both the bedroom difference and the square-foot difference contribute fairly. If you were picking the three nearest neighbours to a new house, the unscaled version would ignore bedroom count and return the three closest mansions, while the scaled version would balance size and layout. That is the practical difference between a model that looks at the data and one that looks only at the biggest numbers.
What does this print? Min-max scale the list [10, 30, 50] to [0, 1].
def min_max_scale(data):
lo = min(data)
hi = max(data)
return [(x - lo) / (hi - lo) for x in data]
scaled = min_max_scale([10, 30, 50])
print([round(x, 2) for x in scaled])
min=10, max=50, range=40
30 is exactly halfway between 10 and 50
Write min_max_scale(data) that returns a new list with every value scaled to the range [0, 1] using the formula (x - min) / (max - min). If all values are identical, return a list of zeros to avoid division by zero.
def min_max_scale(data):
# return [(x - min) / (max - min) for each x]
pass
Compute lo and hi first.
Guard against lo == hi to avoid division by zero.
def min_max_scale(data):
lo = min(data)
hi = max(data)
if lo == hi:
return [0.0 for _ in data]
return [(x - lo) / (hi - lo) for x in data]
Write standard_scale(data) that returns a new list where each value is shifted by the mean and divided by the standard deviation. Use the population standard deviation: sqrt(sum((x - mean)^2) / n). If the standard deviation is zero, return a list of zeros.
import math
def standard_scale(data):
# subtract mean, divide by std dev
pass
Compute the mean first, then the squared differences from it.
Divide the sum of squared differences by n before taking the square root.
import math
def standard_scale(data):
n = len(data)
m = sum(data) / n
s = math.sqrt(sum((x - m) ** 2 for x in data) / n)
if s == 0:
return [0.0 for _ in data]
return [(x - m) / s for x in data]
This function scales test data using the GLOBAL minimum and maximum of the combined dataset. That leaks test information into the training pipeline. Fix it so it scales using only the TRAINING statistics, then applies those same statistics to the test data.
def scale_test_data(train, test):
combined = train + test
lo = min(combined)
hi = max(combined)
return [(x - lo) / (hi - lo) for x in test]
Use min(train) and max(train), not min(combined) and max(combined).
Guard against lo == hi to avoid division by zero.
def scale_test_data(train, test):
lo = min(train)
hi = max(train)
if lo == hi:
return [0.0 for _ in test]
return [(x - lo) / (hi - lo) for x in test]
Write scaled_distance(features_a, features_b, train_data) that computes the Euclidean distance between two houses after min-max scaling every feature using statistics from train_data only. Each house is a list of raw feature values, and train_data is a list of houses. Return the distance as a float.
import math
def scaled_distance(features_a, features_b, train_data):
# 1) transpose train_data to get per-feature lists
# 2) scale each feature of a and b using train stats
# 3) return Euclidean distance between scaled vectors
pass
Build col by collecting house[i] for every house in train_data.
Scale both feature vectors with the same lo and hi, then compute Euclidean distance.
import math
def scaled_distance(features_a, features_b, train_data):
n_features = len(features_a)
scaled_a = []
scaled_b = []
for i in range(n_features):
col = [house[i] for house in train_data]
lo = min(col)
hi = max(col)
if lo == hi:
sa = 0.0
sb = 0.0
else:
sa = (features_a[i] - lo) / (hi - lo)
sb = (features_b[i] - lo) / (hi - lo)
scaled_a.append(sa)
scaled_b.append(sb)
return math.sqrt(sum((x - y) ** 2 for x, y in zip(scaled_a, scaled_b)))
Recap
- Min-max scaling squeezes data into [0, 1] with
(x - min) / (max - min). Use it when bounds are meaningful, like pixel values. - Standard scaling centres on 0 with a spread of 1 using
(x - mean) / std. Use it for gradient descent and whenever outliers should not be compressed against a ceiling. - Scaling does not add or remove information; it changes the geometry so distance-based and gradient-based algorithms treat every feature fairly.
- The golden rule is split first, scale second. Compute statistics on the training set only, then apply them to the test set. Scaling the whole dataset before splitting leaks test data into training — the classic leakage trap you will explore next.