Diffusion: generating by removing noise

Image generators start every picture from pure static: random noise, different every time. Then, in a few dozen small steps, the static turns into a face or a landscape. Nothing in the noise says "face"; the knowledge is in a network that has learned one narrow skill: looking at something noisy and guessing which noise was added.

This lesson shows the whole idea on the smallest possible example: clouds of 2D points instead of pictures, so you can watch every point move.

forward process

Adding a little Gaussian noise at each step $t = 1, \dots, T$, until the data is indistinguishable from pure noise.

noise schedule

How much noise each step adds, $\beta_t$. Its running product $\bar\alpha_t = (1-\beta_1)\cdots(1-\beta_t)$ is how much of the original signal survives to step $t$.

denoiser

A neural network that takes a noisy point $x_t$ and the step $t$ and predicts the noise $\varepsilon$ in it.

reverse process

Generating: start from fresh noise and repeatedly remove a share of the predicted noise, from $t = T$ down to $1$.

The forward process needs no learning at all. Because Gaussian noise adds up to Gaussian noise, you can jump straight to any step:

As $t$ grows, $\sqrt{\bar\alpha_t}$ (the signal) falls towards 0 and $\sqrt{1-\bar\alpha_t}$ (the noise) rises towards 1, so every shape ends as the same blob: a standard normal around the origin.

Training is a guessing game built on that formula: take a data point, pick a random $t$ and a random $\varepsilon$, compute $x_t$, and ask the network for $\hat\varepsilon$. The loss is $\lVert \hat\varepsilon - \varepsilon \rVert^2$. Our denoiser here is a small network with two hidden layers of 48 units, trained once when the site was built.

Before each step, predict where the cloud or the followed point goes next. In the reverse process, use the dashed arrow.

When the noise is large, the guess $\hat x_0$ is poor: a point that is mostly noise could have come from almost anywhere on the shape, so the best guess is a blurry average of all of them, near the middle. Jumping straight to it would land every point in the same place. Instead, each step moves only a little towards the guess and adds a little fresh noise back; at the next step the point is slightly less noisy and the guess slightly better.

Try it with the controls below:

• Change the seed: the generated points are different every time, yet they always land on the shape. They are new samples, not copies of the training points.
• Watch the arrow on the ring: early on it swings around; near the end it is short and steady.
• Compare the shapes: the thin spiral is the hardest, and a few points end up between its arms.

• The forward process adds noise until data becomes $\mathcal{N}(0, I)$; the closed form jumps to any step.
• A network is trained to predict the added noise $\varepsilon$ from $x_t$ and $t$.
• Generating runs backwards from fresh noise: remove a share of the predicted noise, add a little fresh noise, repeat.
• Many small steps are needed because the denoiser's guess only sharpens as the noise fades.
• Image models do exactly this to every pixel at once, with a far bigger network that also reads a text prompt.