Derivatives and gradients: shrink the gap

A car's speedometer reads 50 km/h. Speed is distance over time, yet the needle shows a speed at a single instant, when no time passes. What can that number mean?

Machine learning asks the same question. A model's loss depends on millions of weights, and training asks of each: if this weight grew a tiny bit, how fast would the loss change, and which way? For one input the answer is a derivative; for all of them at once, the gradient. Both come from one move: measure a slope across a small gap, then let the gap shrink.

secant slope

The slope of the line through two points of a curve, $P = (a, f(a))$ and $Q = (a + h, f(a + h))$: rise over run.

derivative f′(a)

What the secant slopes settle on as $h$ shrinks to 0 from both sides: the curve's slope at the single point $a$.

tangent line

The line through $P$ with slope $f'(a)$: the closest straight fit to the curve near $P$.

partial derivative ∂f/∂x

The slope as $x$ changes while every other input is held fixed.

gradient ∇f

All the partial derivatives stacked into a vector. It points the way $f$ rises fastest.

$P$ sits on $f(x) = x^2$ at $a = 1$. A slope needs two points, so take a second point $Q$ a gap $h$ further along, measure the slope between them, and shrink $h$.

For $x^2$, algebra shows why:

At $a = 1$ that is $2 + h$: every secant misses the tangent's slope by exactly $h$. As $h$ shrinks to 0 from either side, $2a + h$ settles on $2a$, so the derivative of $x^2$ is $2x$. In general,

and the limit must be the same from both sides. That is the needle's number: average speeds over shorter and shorter intervals, settling on one value.

Done at every point, this makes $f'$ a function of its own, the Slope f′(x) graph above. Its sign says which way $f$ is going: negative left of 0, where $x^2$ falls, positive right of it. It is 0 where $f$ levels off, at peaks and valleys. Its size says how sensitive $f$ is: at $x = 2.7$, $f' = 5.4$, so a small step in $x$ moves $f$ about 5.4 times as far; near 0, $f$ hardly moves.

Now $P$ sits at the corner of $f(x) = |x|$, at 0. Predict how the secant slope changes as $h$ shrinks, whether a secant from the left agrees, then whether $f$ goes uphill or downhill as the sweep moves along.

A computer can't let $h$ reach 0, so it stops at a small $h$ and divides. This program measures the forward difference, the secant slope with $Q$ to the right of $P$, for $x^2$ at $a = 3$ with $h$ = 0.5, 0.25 and 0.125.

def f(x):
    return x * x

def secant_slope(a, h):
    return (f(a + h) - f(a)) / h

a = 3
h = 0.5
while h >= 0.125:
    print(secant_slope(a, h))
    h = h / 2

Points on both sides of $a$ do better. The central difference, $\frac{f(a + h) - f(a - h)}{2h}$, averages the forward and backward slopes. On $x^2$ those are $2a + h$ and $2a - h$, so the errors cancel exactly; in general what is left is proportional to $h^2$. For $e^x$ at 0 (slope 1) with $h = 0.01$, forward gives 1.0050 and central 1.0000167.

So why not take $h = 10^{-20}$? Because floats have gaps (see floating point). For $x^2$ at 3 in float64, $h = 10^{-8}$ gives 5.99999996, $h = 10^{-12}$ gives 6.0005, and $h = 10^{-16}$ gives 0: floats near 3 are about $4 \times 10^{-16}$ apart, so $3 + 10^{-16}$ rounds back to 3. Gradient checking tests backpropagation this way: nudge each weight about $10^{-5}$ either way and compare the central difference of the loss with the backward pass's answer.

Nobody works out every derivative from the limit. A few rules, each proved once from it, cover this lesson's functions:

• Power rule: $x^n$ gives $n x^{n-1}$, so $x^3$ gives $3x^2$. A constant gives 0.
• Constant multiple and sum: $(c f)' = c f'$ and $(f + g)' = f' + g'$, so $x^3/3 - x$ gives $x^2 - 1$.
• Two special functions: $e^x$ is its own derivative, and $\sin x$ gives $\cos x$ (in radians).

The rules assume the curve is smooth where you use them.

A loss depends on many weights at once. For $f(x, y)$, the partial derivative $\partial f/\partial x$ is the ordinary derivative in $x$ with $y$ held fixed like a number, and $\partial f/\partial y$ the other way round. In the narrow valley $f(x, y) = x^2/20 + 5y^2$ they are $x/10$ and $10y$. Stacked, they make the gradient $\nabla f = (x/10, 10y)$, which at $(-3.5, 1.5)$ is $(-0.35, 15)$.

Walk in the direction of a unit vector $u$, and the slope underfoot is the dot product $\nabla f \cdot u$. That is largest when $u$ points along $\nabla f$, so the gradient points the way $f$ rises fastest, and its length is that steepest slope: about 15 at $(-3.5, 1.5)$, where walking along $+x$ gives just −0.35. At right angles to $\nabla f$ the slope is 0, which is along the contour line: the gradient crosses contour lines at right angles.

The tangent guess works in many dimensions too: $f(p + \Delta) \approx f(p) + \nabla f \cdot \Delta$ for a small step $\Delta$. Step by $\Delta = -\eta \nabla f$ and the guess falls by $\eta\,|\nabla f|^2$, so a small enough step against a nonzero gradient always goes downhill. Gradient descent is that one idea, repeated.

• $f'(a)$ is the limit of the secant slope as $h \to 0$, from both sides. At a corner, like $|x|$ or ReLU at 0, the sides disagree and there is none.
• The sign of $f'$ says rising or falling, it is 0 at smooth peaks and valleys, and its size says how sensitive $f$ is. Near $a$, $f(a + h) \approx f(a) + f'(a)\,h$.
• Rules: $x^n$ gives $n x^{n-1}$, constant factors stay, sums split, $e^x$ stays $e^x$, and $\sin x$ gives $\cos x$.
• The forward difference's error is proportional to $h$, the central difference's to $h^2$, and too small an $h$ loses digits to rounding.
• The gradient stacks the partial derivatives. It points the way $f$ rises fastest, its length is that slope, it crosses contour lines at right angles, and the slope along a unit vector $u$ is $\nabla f \cdot u$.
• Next: gradient descent walks against the gradient, and backpropagation computes it for every weight. Then try Backpropagate through a neuron by hand.