Every feature in your dataset is one axis. Two features plot as a flat sheet, three as a cube you can still picture, but a customer record with fifty fields lives in a fifty-dimensional space no human can visualise. Adding features feels like adding information, yet past a point each new axis spreads the same number of points across exponentially more room, until your data is mostly empty space and the tools that depend on points being meaningfully near each other, like nearest-neighbour search and k-means, quietly stop working. Often the features are redundant too, several carrying the same underlying signal in different units, so the true number of independent directions is far smaller than the column count. Dimensionality reduction is the answer to a problem that more features created.
flowchart LR O["object in 3D"] -->|"light casts a shadow"| S["2D silhouette"] S --> K["keeps the outline"] S --> L["loses depth:<br/>different objects can share one shadow"] style O fill:#0e7490,color:#fff style S fill:#b45309,color:#fff
A projection is a dot product
The mechanism is simple. Pick a direction, described by a vector with one number per feature, and for each point compute the dot product, the sum of feature-times-direction products. That single number is the coordinate of the point along that direction, just as its shadow falls at one spot on the wall. Keep a few well-chosen directions and you have mapped every point from many axes down to a few, while throwing away the directions you did not keep. Nothing is invented; information is only discarded, and the whole art is choosing directions whose loss hurts least.
def project(point, direction):
return sum(a * b for a, b in zip(point, direction))
for p in [[2, 0], [0, 2]]:
print(project(p, [1, 1]))
Look closely: the points [2, 0] and [0, 2] are far apart in two dimensions, but both project to the value 2 onto the diagonal direction. The projection collapsed two genuinely different points onto the same number. That collision is information loss made literal, and it is unavoidable whenever you map a higher-dimensional space onto fewer axes. The only question is whether the discarded detail mattered, which depends entirely on which directions you keep and which you drop.
flowchart TD H["more dimensions"] --> V["volume grows exponentially"] V --> SP["points become sparse"] SP --> C["all pairwise distances concentrate<br/>near the same value"] C --> F["nearest and farthest look alike:<br/>k-NN and k-means degrade"] style H fill:#0e7490,color:#fff style F fill:#b91c1c,color:#fff
import random, math
def distance(a, b):
return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))
def spread(n, dim, seed):
rng = random.Random(seed)
pts = [[rng.random() for _ in range(dim)] for _ in range(n)]
ds = []
for i in range(len(pts)):
for j in range(i + 1, len(pts)):
ds.append(distance(pts[i], pts[j]))
lo, hi, mean = min(ds), max(ds), sum(ds) / len(ds)
return round((hi - lo) / mean, 3)
for dim in (2, 50, 500):
print('dim', dim, 'relative spread', spread(150, dim, seed=7))
The printed relative spread is the gap between the largest and smallest pairwise distance, divided by the average distance. In two dimensions it is a healthy number because some points land close together and others far apart. As the dimension climbs to fifty and then five hundred, that ratio shrinks toward zero: in a high-dimensional cube, essentially every pair of random points sits at nearly the same distance from every other pair. When everything becomes equidistant, nearest-neighbour search loses its meaning, distance-based clustering cannot separate groups, and the data becomes impossible to plot. Your fifty-feature model also has more parameters than a few hundred rows can support, which invites overfitting on top of the sparsity.
def project(point, direction):
return sum(a * b for a, b in zip(point, direction))
def variance(values):
m = sum(values) / len(values)
return sum((v - m) ** 2 for v in values) / len(values)
data = [[3, 0], [4, 0], [5, 0], [6, 0]]
print('variance along x:', round(variance([project(p, [1, 0]) for p in data]), 2))
print('variance along y:', round(variance([project(p, [0, 1]) for p in data]), 2))
The data runs left to right and is flat top to bottom, so projecting onto the x-axis keeps almost all the spread and projecting onto the y-axis keeps almost none. For this tidy, linear, axis-aligned data, keeping the high-variance direction is clearly the right call. Hold that success in mind alongside the warning above: variance-tracking only obviously wins when the spread happens to line up with the signal. The moment your structure is bent or the noise is loud, the loudest direction stops being the most informative one, and a linear projection can flatten exactly the shape you needed, just as k-means flattened the rings in the previous lesson.
flowchart LR D["pick directions to keep"] --> HV["highest-variance direction"] HV --> Q["is it signal or noise?"] Q -->|"loud flickering sensor"| N["high variance, zero signal"] Q -->|"quiet class separator"| S["low variance, all signal"] style N fill:#b91c1c,color:#fff style S fill:#166534,color:#fff
Write project(point, direction) returning the dot product: the sum of each coordinate of point multiplied by the matching coordinate of direction.
def project(point, direction):
# sum of point[i] * direction[i]
pass
Pair up coordinates with zip.
Multiply each pair and sum the results.
def project(point, direction):
return sum(a * b for a, b in zip(point, direction))
What does this print? Two points are projected onto the diagonal direction.
def project(point, direction):
return sum(a * b for a, b in zip(point, direction))
for p in [[2, 0], [0, 2]]:
print(project(p, [1, 1]))
project([2, 0], [1, 1]) is 21 + 01.
project([0, 2], [1, 1]) is 01 + 21.
This project ignores the direction entirely and just sums the point's coordinates, so the projection does not depend on which direction you chose. A projection must multiply each coordinate by the matching direction weight. Fix it.
def project(point, direction):
return sum(a for a, b in zip(point, direction))
Each term must multiply the point coordinate by the direction weight.
Change
atoa * b.
def project(point, direction):
return sum(a * b for a, b in zip(point, direction))
Write reconstruction_error(points, projected_values, direction) measuring what a projection LOSES. For each point, rebuild it from its projected value as projected * direction, then add the squared distance between the original point and that rebuild. direction is a unit axis such as [1, 0]. The helper reconstruct is given to you.
def reconstruct(projected, direction):
return [projected * d for d in direction]
def reconstruction_error(points, projected_values, direction):
# for each point, add squared distance to reconstruct(its projected value, direction)
total = 0
pass
return total
Rebuild each point with reconstruct(projected_value, direction).
Add the squared coordinate differences between the original and the rebuild.
def reconstruct(projected, direction):
return [projected * d for d in direction]
def reconstruction_error(points, projected_values, direction):
total = 0
for p, pv in zip(points, projected_values):
rec = reconstruct(pv, direction)
total += sum((p[j] - rec[j]) ** 2 for j in range(len(p)))
return total
PCA keeps the directions of greatest variance. Is the highest-variance direction always the most informative for your task?
Variance measures spread, not relevance. A flickering sensor or bright-light channel varies hugely but carries no signal, so maximising variance retained can preserve noise and discard exactly the low-variance direction that separates your classes.
Recap
- More features means more room: each added axis spreads the same points across exponentially more volume, leaving them sparse.
- Distances concentrate in high dimensions: nearest and farthest neighbours grow similarly far apart, so distance-based methods like k-NN and k-means degrade.
- A projection maps points to fewer axes via a dot product onto chosen directions; it is lossy, like a shadow that keeps the outline but drops the depth.
- It preserves variation along the kept directions and discards the rest; the lost detail shows up as reconstruction error and as collisions where different points share one coordinate.
- Variance is not information: keeping the loudest direction is a sensible default, never a proof of relevance, and a linear projection can flatten the very structure you needed.
That closes the unsupervised arc: you can now find groups when given no labels, see when those groups are lying to you, and compress a feature space down to the dimensions that actually carry signal.