Rolling downhill
How a marble finds the bottom of a hill, and how computers learn the same way.
What you are looking at
The dots flow across a bumpy landscape and collect in the low spots. Some low spots collect a lot of dots, and some collect few.
The idea
Put a marble on a bumpy table. It rolls downhill and stops in the first dip it finds. It does not look around for the deepest dip. It only feels which way is down, right where it is.
That simple rule, "go down from here," is very useful. It finds a low spot without needing a map of the whole table. The catch is that the low spot you find may not be the lowest one. A deeper dip may be on the other side of a small hill.
Computers use this same rule to learn. They measure how wrong their answers are, which is like a height, and they take small steps in the direction that makes the error go down.
Where else you see it
- Rain running downhill into streams and lakes.
- Finding the lowest point of a field by walking downhill in the fog.
- A phone learning to recognize your voice by making a lot of tiny adjustments.
Go deeper: the proper names and the math
This is gradient descent: x ← x − η ∇f(x), where η is the step size (learning rate) and ∇f points uphill. It converges to a local minimum, and on a surface with many dips there is no guarantee it is the global one. Too large a step overshoots, and too small a step takes forever.
Training a neural network means descending a surface with millions or billions of dimensions. In practice, adding randomness (stochastic gradient descent) and momentum helps the process avoid getting stuck in shallow dips and flat spots.