How AI learns from mistakes: walking downhill in the fog
Big idea: a model learns by taking small steps downhill on its errors, again and again.
The story in short
On a foggy dawn trek on Nandi Hills, nobody could see the path. So they felt which way the ground sloped and stepped downhill. Happy took a giant leap and overshot. Bunny took baby steps and was very slow. They got stuck in a hollow with a little pond that looked like the bottom, until Mitthu flew up and saw the real valley below. Back home, Bunny walked his cricket neuron’s knee weight the same way: −1 gave 9 wrong Sundays, −2 gave 6, −3 gave 4, −4 gave 5. So he kept −3, and the neuron got 146 of 150 Sundays right.
The idea
- Error (loss): a number for how wrong the model’s answers are. Height on the hill.
- Weights: where you are standing.
- Gradient (slope): which way each weight should move to reduce the error.
- Gradient descent: step each weight a little downhill, again and again. Most neural networks are trained this way.
- Learning rate: how big each step is. Too big overshoots; too small is slow.
- Local minimum: a low point that isn’t the lowest point.
- Backpropagation: the method networks use to find the slope for every weight at once.
Remember: you don’t have to see the whole way. Feel where you are, take a small step in the right direction, and keep correcting.
Check yourself
Bunny tried knee weights −2, −3 and −4 and got 6, 4 and 5 wrong. Which weight should he keep?
Training is very slow because each step is tiny. What would help?
In gradient descent, what does the "height" on the hill stand for?
More for you: 🎒 Students · 🧑🏫 Parents and teachers