Gradient descent: how models learn

01Why it matters

How a model learns: walk downhill

Training a neural network means finding the weights that make its error, the loss, as small as possible. There can be billions of weights, so trying them all is hopeless. Instead, start somewhere, feel which way is downhill, take a small step, and repeat.

That is gradient descent, and almost every model you have heard of was trained by some version of it. To use it you need one idea from calculus, the slope, and one decision, how big a step to take.

02Key parts

The pieces you will see

derivative f′(x): The slope of $f$ at a single point: how much $f$ changes per unit step in $x$ , for a tiny step.
gradient ∇f: The slope in every direction at once, as a vector. It points the way $f$ rises fastest.
step size η: "Eta", also called the learning rate: how far to move against the gradient each step.
contour lines: Lines of equal $f$ on a map, like height lines on a hiking map. The gradient crosses them at right angles.

03See it work

First, the slope at a point

A slope needs two points. Take $P$ and a second point $Q$ a gap $h$ away, measure the slope of the line through them, and shrink $h$ : the slopes settle on one number, the derivative.

The secant through P and Q turns into the tangent as h shrinks. Then the slope is measured everywhere to draw f′.

f'(a) = \lim_{h \to 0} \frac{f(a + h) - f(a)}{h}

In two dimensions, do the same along $x$ and along $y$ and stack the two slopes into a vector: the gradient $\nabla f$ . It points uphill, so $-\nabla f$ points downhill, and that gives the whole algorithm:

x \leftarrow x - \eta \, \nabla f(x)

Each step: the gradient (uphill) is measured, and x moves against it. On a round bowl that points straight at the bottom.

04Predict

Your turn: which way is the slope?

Before each step, predict whether $f$ is rising, falling or flat at the next point. That is exactly the information gradient descent uses.

Make 6 predictions: step forward and answer each question before the next step plays.

05Try it

Choose the step size

Pick an example and drag the start point. Things to try:

Too big a step: with $\eta = 1.05$ on $x^2 + y^2$ , every step overshoots further than the last. Does $f$ ever fall?
Narrow valley: the step that suits the gentle slope along the valley is too big for its steep sides, so the path zigzags.
Tiny steps: safe, but 40 steps barely move.
Two valleys: start on the right and descent settles in the higher valley. Drag the start to the left half.

06Try it

Better steps: momentum and Adam

Plain gradient descent forgets everything after each step. Momentum keeps a running velocity, so steps that agree build up speed and steps that zigzag cancel out. RMSProp divides each direction's step by how steep that direction has recently been, so steep and gentle directions move at similar speeds. Adam, the default for training most neural networks, does both.

Four optimizers, one start, the same gradients. Plain descent crawls along the curved valley floor; momentum keeps moving.

07Check yourself

Check yourself

Question 1 of 4
For f(x) = x², what is the slope of the secant from a = 1 to a + h?
Your answer comes back in Review in a few days, so it sticks.
Question 2 of 4
The gradient at x is (3, −4). Which way does gradient descent step?
Question 3 of 4
f went up after a step of gradient descent. What is the likeliest fix?
Question 4 of 4
Why does momentum help in a long, narrow valley?

08Explain it back

Say it in your own words

How does gradient descent find a low point of a function, and why does the step size matter so much?

09Recap

What to remember

The derivative is the limit of secant slopes; the gradient collects the slopes in every direction and points uphill.
Gradient descent repeats $x \leftarrow x - \eta \nabla f(x)$ until the ground is flat.
The step size matters most: too small crawls, too big overshoots or diverges, and narrow valleys make plain descent zigzag.
It finds a local minimum. Momentum, RMSProp and Adam add memory to move faster and more evenly.