Machine learning · 15 min

Linear regression: shrink the squares

A model, a loss and gradient descent: fit a line by shrinking squared residuals you can see, and drag the data.

Builds onGradient descentVectors and matrices

01Why it matters

The first model everyone trains

Given some points, find the straight line that fits them best. It sounds like school geometry, but it is a complete machine-learning pipeline in miniature: a model (y=wx+by = wx + b), a loss that says how wrong it is, and gradient descent nudging the parameters until the loss stops falling.

Every bigger model, up to large language models, is trained with the same three pieces.

02Key parts

The pieces you will see

model
The line y^=wx+b\hat{y} = wx + b: a weight ww (the slope) and a bias bb (where it crosses zero).
residual
How far a point is from the line vertically: y−y^y - \hat{y}.
mean squared error
The average of the squared residuals. Here each one is drawn as an actual square, so the loss is their average area.
least squares
The line with the smallest possible MSE, which also has an exact formula.

03See it work

Shrink the squares

Each step moves w and b against the gradient of the MSE: the squares shrink and the line swings towards the data.

The gradient of the MSE comes straight from the chain rule: each point pulls the line towards itself in proportion to its residual,

∂ MSE∂w=−2n∑i(yi−y^i) xi,∂ MSE∂b=−2n∑i(yi−y^i).\frac{\partial\,\mathrm{MSE}}{\partial w} = -\frac{2}{n}\sum_i (y_i - \hat{y}_i)\,x_i, \qquad \frac{\partial\,\mathrm{MSE}}{\partial b} = -\frac{2}{n}\sum_i (y_i - \hat{y}_i).

04Predict

Your turn: which way will it move?

Make 5 predictions: step forward and answer each question before the next step plays.

0 of 5 made

05Try it

Drag the data

Drag any point and the fit runs again from the start. Try:

  • Drag one point far from the others. Watch its square dominate the loss and drag the line towards it.
  • Line the points up exactly. The final MSE goes to 0.
  • Pick "Too big a step": with too large a learning rate, the line swings further each time and the loss explodes.

06Check yourself

Check yourself

  1. Question 1 of 3

    Three residuals are 1, −2 and 3. What is the mean squared error?

    Your answer comes back in Review in a few days, so it sticks.

  2. Question 2 of 3

    Every point lies above the line. Which way does gradient descent move b?

  3. Question 3 of 3

    Why can one outlier move the least-squares line so much?

07Explain it back

Say it in your own words

What makes one line fit the data better than another, and how does gradient descent find the best one?

08Recap

What to remember

  • A model, a loss and an optimiser: y^=wx+b\hat{y} = wx + b, the mean squared error, and gradient descent.
  • Each point pulls the line in proportion to its residual; squared errors make outliers pull hard.
  • Gradient descent reaches the same line as the exact least-squares formula when the step size is right.
  • Next: replace the line with a curve made of many neurons, and the same recipe trains a neural network.
Back to path