Skip to main content
Loading Explore guides…
Diffusion Models · Acedra
Home
Explore
Computer Science
Diffusion Models
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
67 segments
Copy
Hide video
0:00
Diffusion models are now used in just about every domain you can think of: generating images, videos, music, 3D models — and even discovering new molecules. But while the results are very impressive, the underlying theory can feel a bit, well… dense. There are lots of important details, and the math isn’t always friendly at first glance. We’ll try to untangle the key ideas behind diffusion models by following step by step the research paper that made them so popular.
0:36
The paper that brought diffusion models into the spotlight is called "Denoising Diffusion Probabilistic Models", or DDPM, by Jonathan Ho and collaborators, and it was published in 2020. Despite its reputation, this paper did not introduce diffusion models, nor did it coin the term. The earliest known reference to diffusion models as we know them comes from a 2015 paper titled "Deep Unsupervised Learning using Nonequilibrium Thermodynamics",
1:03
by Jascha Sohl-Dickstein and colleagues. In 2015, GANs were considered state of the art in image generation research, and diffusion models received little attention. In contrast, the 2020 DDPM paper quickly gained traction—it is now the most cited work in the field, with more than twenty thousand citations as of 2025. If you look up diffusion models on any search engine, this is probably the first paper you'll come across. The main contribution of this paper was to
1:37
take diffusion models from a theoretical concept—previously limited to simple toy datasets—into a state-of-the-art technique for image generation. On the technical side, it mainly introduced a much simpler training objective compared to the 2015 version. This made the method more accessible, and highlighted interesting connections to earlier works on denoising models. Now, let’s break down the core idea. In the context of data generation,
2:05
we start with a probability distribution p of x, which represents our training data. For instance in the case of images, we can think of this distribution as representing all natural images, or even just a subset of natural images containing only faces. Of course, this distribution is so complex that we can't really find a single expression to describe it completely, and there are probably no such expression anyway.
2:32
But even without an explicit formula, we still want to generate new images, which is equivalent to sampling new points from this underlying distribution. So the challenge is to find a way to create new samples without actually having a good, comprehensive, way of describing the distribution. Diffusion models tackle this problem with a seemingly unrelated process — by removing Gaussian noise from images. First, we gradually corrupt the images by adding Gaussian noise.
2:59
At each step, a small amount of noise is added, slowly erasing the structure of the image. If we continue this process for many steps, the image eventually turns into pure noise. The key idea is then to train a model that learns to slowly reverse these steps, and remove the noise bit by bit. If the model is effective, it should be able to start from random noise and refine it step by step into a meaningful image. Now, let’s see how the DDPM paper formulates this process mathematically.
3:29
Let's begin with an image, which we'll call x-zero, since it's the starting point of the noising process. For now this image does not contain any noise. To generate the next image, x-one, we use a conditional distribution: q of x-one given x-zero. We produce x-one by adding some amount of Gaussian noise epsilon. The amount of noise is controlled by a scalar value, beta. Then actually the distribution describing x-one is just a Gaussian centered on
3:58
x-zero with noise of variance beta. So rather than seeing this as adding noise to x-zero, we can just imagine that we take a sample from a Gaussian distribution of variance beta, centered on x-zero. Then, to go to the next step, we do the same thing. This time we label the distribution as q of x-two knowing x-one. Again the operation is simply taking the previous step and adding some amount beta of Gaussian noise. Which is the same as taking a sample from a Gaussian distribution
4:31
centered on x-one with variance beta. Now since we just sum the noise terms, we can go straight from x-zero to x-two. We created x-two by adding twice as much noise, which is the same as sampling from a Gaussian centered on x-zero, with variance two beta. Now we keep repeating this process, step by step, until we reach the timestep t. If we think a little, we can write a simple formula to describe what happens at any given step: we simply add t times beta amount of Gaussian noise to x-zero.
5:03
So at timestep t, x-t knowing x-zero is a Gaussian distribution centered at x-zero, with variance t-beta. Now we have an issue with this process. What we wanted was a process that gradually transforms our data into a standard normal distribution; clearly, that's not what's happening here. The mean always stays fixed at x-zero and is never modified by the process, while the variance keeps increasing indefinitely. So in this setup, there is absolutely
5:30
no chance that our diffusion process converges to the normal distribution. In this diffusion process, the variance just keeps growing without bound: in the litterature we often say that the variance explodes; and we call this process the variance exploding diffusion. In the DDPM paper, the authors made the choice to use a noising process that converges to a normal distribution, which is often called variance preserving, so let's see how they do it.
5:55
Let’s now define a diffusion process that actually works — one that gradually transforms our data into a standard normal distribution. For this, we need to satisfy two conditions: the mean has got to converge to zero and the variance to one. To do that we'll modify the way we go from a given step to the next. Obviously, we'll need to change the mean of our distribution at each step. And so we'll need a coefficient in front of the mean to make it decrease to zero.
6:25
This magic number that we will use is square root of one minus the variance beta. Hm well, why do we choose this number in particular ? To be honest, multiple other valid choices exist — however, the authors of the DDPM paper specifically opted for this coefficient. The main reason for this choice is probably its simplicity. I'll skip the derivations because this is quite honestly pretty boring, but the important thing is that this choice results in
6:52
a straightforward and convenient expression for the conditional distribution at any time step. So to go from the noiseless input x-zero, to a given time step of our diffusion process, we just have to apply this little formula. It depends on the term alpha-t bar, which is simply defined as one minus beta, raised to the power of t. Now does this thing actually converge to the normal distribution ? Well if we choose beta strictly between zero and one, it is easy to see that alpha-t-bar
7:23
will converge to zero as t grows to infinity. Then the mean of the conditional distribution goes to zero, while the variance goes to one. And just like that, we finally have a proper variance preserving diffusion process. So, this formula is supposed to take our distribution of data p of x, and slowly transform it into a normal distribution: let's see this on a simple 1D example. Let's say this is our original noiseless data distribution p of x. After applying just one diffusion step,
7:55
the distribution barely changes. However, if we repeat this process for about a hundred steps, we can clearly see the effect: the mean approaches zero, and the spread of the distribution stabilizes. And the nice thing is that we can go directly from the input image to a given time-step ! Now, if you've explored the DDPM paper yourself, you might have noticed they use a slightly different definition for alpha-bar. In practice, diffusion models commonly apply varying amounts of
8:28
noise at each step, rather than a fixed beta. Indeed, any noise schedule can be chosen as long as each beta remains between zero and one. How does that fact impact our formula to go from the noiseless image to any timestep ? Well when beta varies at each step, alpha-bar at step t simply becomes the product of one minus beta at every previous step, which is the expression you can find in the paper. So we have now defined a proper diffusion process
8:56
that will progressively transform any distribution into a normal distribution. Let's now move onto how we can reverse this diffusion process using a neural network. Just for the sake of illustration, let's represent once again the forward process that enables us to go from our clean image 'x-zero' to a Gaussian noise sample 'x-T', using the probability densities q. Now let's assume there is a way to reverse these individual steps, and let's represent it with a probability density p,
9:27
which produces x-t-minus-one given x-T. So, just like we defined conditional distributions q to go forward from one step to the next, we now define conditional distributions p to go backward in the opposite direction. While in the forward process all parameters are fixed, for the reverse process our goal is actually to find the best parameters theta that allow us to remove the noise effectively. And as you've probably guessed, these parameters
9:54
theta will be the weights of a neural network. Now you may ask: how do we train this neural network? Well, we'll adopt a very standard approach in Bayesian statistics, which is to minimize the negative log likelihood of producing a sample with our model. If you're not familiar with these types of objectives, don't worry. It simply means finding the set of parameters theta that maximizes the likelihood of generating real samples x-zero from our
10:19
data distribution, using the neural network. But instead of maximizing p theta of x-zero directly, we minimize the negative log likelihood, which in practice is easier in many cases. Ok, before we go further, let's define some useful notations clearly. The joint probability describing the distribution of all the variables from x-one to x-T given x-zero represents our complete forward process. Using Bayes' theorem, we can decompose this joint
10:49
probability into individual conditional distributions. And the neat thing is that since our forward process is a Markov chain, each noising step depends only on the previous step, which simplifies these conditional probabilities. Another notation we use is that instead of writing three dots every time we want to mention the complete forward process, we use this simplified version with two dots. And now let's rewrite this product in an even more compact form,
11:16
using the standard notation for products. We use the same compact notation for the joint distribution describing our complete reverse process. As you may have noticed, the reverse process is not conditioned on anything: indeed, we will start from pure Gaussian noise, without prior knowledge, whereas the forward process requires an input image x-zero. Ok, now that we have expressions describing clearly our complete forward and reverse processes, let's try to find an expression
11:46
for the negative log likelihood. Generally speaking, when dealing with joint probabilities, a useful first step is often to marginalize the distribution with respect to the other variables. Doing this gives us that p theta of x-zero is simply the integral of the joint probability with respect to all the other variables. Now could we compute the likelihood using just this formula? Hmm well, no; not really. What this integral means is that to
12:15
compute the probability that our neural network generates a sample x-zero, we would have to sum over all possible paths capable of generating it. Let's visualize it this way: on the left we have the density of natural images, and on the right the pure Gaussian noise density. We know that there are paths connecting images to noise, such as our forward process, and we assume that there exist paths connecting noise to images. The formula tells us that we should sum over all
12:44
these possible paths to obtain the negative log likelihood. Of course, there are way too many possible paths for us to compute this quantity. We say this quantity is intractable, meaning we can't realistically compute it exactly. Ok, but that does not mean we can't compute the negative log likelihood at all! Let's now use a very well-known trick: we multiply and divide by the density representing our forward process. We can rewrite this integral as an expectation,
13:13
making the expression slightly more compact. And here comes another very well-known trick: Jensen's inequality. Since the negative log function is convex, we can put the log inside the expectation at the cost of introducing an inequality. And now we obtain an upper bound on our negative log likelihood, often called the evidence lower bound, or ELBO. Why is it called a lower bound? Well, since it's an upper bound on the negative log likelihood, it corresponds to
13:41
a lower bound on the likelihood itself. Now, we'll try to make sense of this training objective by modifying and simplifying its expression further. I'm going to skip the detailed derivations here, because they're honestly a bit tedious, and the video is already pretty math-heavy. Just know that with some straightforward algebra and Bayes' theorem, we can rewrite the ELBO from this form to this more useful expression. If you're interested in the full computations leading to this expression
14:10
of the ELBO, it's covered in part A of the supplementary materials of the DDPM paper. It's actually quite detailed and worth checking out if you're curious. Now, this new expression might look intimidating at first glance, but as usual, there's some intuition behind it. The first thing you might notice is the Kullback–Leibler divergence term, which we refer to as D–K–L. The Kullback–Leibler divergence is a way of measuring how close two probability
14:37
distributions are — you can think of it as a kind of distance in probability space. If the distributions are very different — for example, they have different means or variances — the KL divergence will be large. On the other hand, if the distributions are very similar, the KL divergence becomes small. And it actually reaches zero only when the two distributions are exactly the same. So, going back to our loss expression,
15:05
we see that it's essentially measuring how far apart certain probability distributions are. Before we see what these distributions are, we can simplify this expression further. The first term on the left doesn’t depend on our model parameters, so we can ignore it when optimizing the model. The last term on the right does depend on the weights of the network, and it measures how likely the model is to reconstruct a
15:29
clean image from a slightly noisier one — more precisely, from x–1 back to x–0. But remember, in diffusion models, we usually take about a thousand denoising steps. So by the time we go from x–1 to x–0, there’s barely any noise left — the model has already recovered most of the image. As a result, this term is pretty small and doesn’t provide much extra learning signal. In practice, including it in the loss makes little to no difference, so we usually drop it from the final loss.
16:00
That leaves us with the main part: a big sum of KL divergences between some distributions. We know that p–theta of x–t–minus–1 given x–t is our reverse diffusion step. But what exactly is this other distribution, q of x–t–minus–1 given x–t and x–0 ? Well, this distribution is what we call the true posterior. If we have our forward process — q of x-t given x-t-minus-one — we know there's a reverse process, which we're trying to approximate, by adjusting the parameters of a distribution p.
16:33
Now, if we also condition on the original image, x-zero, we can actually compute the exact reverse process. This true reverse process, which uses the clean image x-zero, is what we call the true posterior. And the nice thing is, we actually have a closed-form expression for this true posterior. You might be wondering — why do we need a neural network if we already have a closed-form expression for reversing the noise?
16:59
The answer is: using this true posterior is cheating, because it requires us to know x-zero — and during inference, our goal is to generate x-zero starting from pure noise. So of course, we can’t use the true posterior conditioned on x-zero to produce new samples. But we can apply it when x-zero is known, like during training, which is exactly what we're doing here. So what we’re really doing with this loss function is training the neural network to match the true posterior — the one we
17:28
could compute if we had access to x-zero — even though, in practice, the network only sees x-t. Now, about the expression of this true posterior — I’ll skip the derivations again, because it’s just some tedious algebra and Bayes’ rule. In the end, it turns out that the true posterior is Gaussian, with a mean mu-tilde and a variance beta-tilde. Alright, now we need to choose a form for our approximate posterior. Naturally, we’ll make it Gaussian, and there are two reasons for that.
18:00
First, there's a theorem that says the reverse of a Gaussian Markov chain is also Gaussian — so choosing a Gaussian makes sense. And second, everything is just simpler when working with Gaussians. Our neural network will therefore predict the parameters of this Gaussian — its mean, mu-theta, and its variance, sigma-theta, which controls its spread. To simplify things even further, the authors choose to fix the
18:24
variance of the approximate posterior to a constant sigma-t, which is not learned. That means the neural network only has to learn the mean of the distribution. Since we’re trying to match two Gaussian distributions, but can only control the mean of the approximate one, the best we can do is bring its mean as close as possible to the mean of the true posterior. Said differently, we aim to minimize the distance between the two means. And in fact, that’s exactly how the
18:50
KL divergence behaves in this case. As you can see, in this setting, the KL divergence simplifies to a plain squared distance between the means of the Gaussians. So, zooming out: minimizing the sum of KL divergences in our objective function really just means bringing each approximate posterior closer to its true counterpart. And since we can only control the mean of each approximate posterior, we just try to reduce
19:17
its distance to the mean of each true posterior. Now that we have a better intuition of the elbo, we'll simplify it further to state the final training objective. We just saw that each of the KL divergence terms in this sum simplifies into a squared distance — so overall, we’re minimizing a sum of L2 distances. Each of these terms measures how far the predicted-mean mu-theta is from the true posterior-mean mu-tilde, and the distance is weighted by a factor that depends on sigma-t.
19:48
Now let’s simplify this loss even further, to reach the final expression. We do have a closed-form expression for the true posterior mean mu-tilde, but it is a bit complicated. As you can see, it’s a weighted combination of x-zero and x-t, which is not very convenient to work with. Now thanks to the reparameterization trick, the same as we used in the beginning of this video — and the same one used in V-A-Es — we can write x-t as being the sum between
20:14
x-zero and epsilon, a pure Gaussian noise. Conversely, with a tiny bit of algebra, we can also write x-zero as depending only on x-t and epsilon. If we plug this into the expression of mu-tilde, we also get an expression that depends only on x-t and epsilon. Then comes the key idea: since the denoiser also has access to x-t, we can apply the same trick to the network’s predicted mean mu-theta, and we write it as a function of x-t and a predicted noise term epsilon-theta.
20:46
What we did was simply to express mu-tilde and mu-theta with x-t and epsilon by using the reparameterization trick. Now if we plug both of these expression into the loss, the terms in x-t cancel out, revealing our final objective function. We’re left with a much simpler loss: a plain squared distance between epsilon and epsilon-theta. So epsilon-theta becomes the network’s estimate of the noise that was added to x-zero to produce x-t — and minimizing the loss just means
21:18
making this prediction as accurate as possible. Ok great, I think it's time to recap the whole process of how we got to this final loss expression.
21:30
We started by saying we want to optimize the negative log-likelihood of x-zero, which represents the probability that a sample comes from our underlying distribution. Since that’s intractable, we introduced a surrogate called the ELBO — the evidence lower bound. Using Bayes' theorem and a lot of computations, we can express this surrogate as a sum of several terms. One of these terms did not depend on our model's parameters, and the other was negligible,
21:58
so we dropped both of them, which left us with a sum of Kullback-Leibler divergences. Then, by choosing the approximate posterior as being Gaussian, we saw that the KL terms reduce to L-2 distances between the true and estimated posterior means. Next, using the reparameterization trick, we rewrote those means in terms of epsilon — the actual noise — and epsilon-theta, the predicted noise. That gave us our final loss: the squared distance between real and predicted noise.
22:28
And this is how we go from the negative log likelihood, to the ELBO, to a final L2 loss function. However we can do a final simplification; as you can see, summing over hundreds of timesteps for every sample is pretty expansive. Instead of summing over all timesteps, we simply pick a random timestep t for each sample, and we know that for a large number of samples this will converge towards the original objective. That’s it — we now have a very simple and efficient training objective.
22:59
Let’s now see how to implement this in practice, and what the actual training and inference look like. Alright, for this python implementation we’ll mainly rely on two libraries that we'll import right away. The first one is PyTorch — no surprise there — which gives us all the necessary tools to define and train neural networks. The second one is a library called DeepInverse — or 'deepinv' for short.
23:23
It’s built on top of PyTorch and is designed specifically for solving imaging inverse problems, which is what I research in real life. We’ll use it here because it provides a ready-to-use implementation of the neural network architecture used in DDPM, along with several pretrained weights. The first thing we do after importing the libraries is set up the dataset. We’re using MNIST — a dataset of small grayscale images with handwritten digits. We’ll use a batch size of 32 and resize
24:50
each image to 32 by 32 pixels. Next, we define everything related to the neural network and the training setup. With DeepInverse, this is straightforward: we just instantiate the Diffusion UNet class, and specify the number of input and output channels. In our case, we’re training the model from scratch, so we won’t load any pretrained weights. For optimization, we’ll simply use the Adam optimizer with a standard learning rate. And for the loss function, as we discussed earlier in the video,
25:21
we’ll use the simple mean squared error. The next step is to define all the constants that will be used for diffusion. First, the beta schedule — this describes the noise level added at each step of the diffusion process. We’ll use a linear interpolation from a small beta-start to a larger beta-end, just like in the original DDPM paper. We can actually change this interpolation, for instance we can choose a cosine schedule, but, for now, let's keep it simple
25:50
and stick with the usual linear interpolation. From this, we compute the alphas, the cumulative products alpha-bar, and other common terms like the square root of alpha-bar, and its complement.
26:03
Now that the constants are set, we can start the training loop. At each step, we sample a random timestep t between 0 and 1000, and draw a noise sample epsilon from a standard normal distribution. We then compute the noisy image using the variance-preserving formulation we derived earlier. As you can see, the formula we use in theory is exactly the same one used in the actual implementation. This noisy image and the timestep are then fed into the network.
26:32
The model output the estimation of the noise present in the image, and we compute the loss between this prediction and epsilon. Now in the code you see that we don't use the exact loss we derived earlier, but an even simpler version. We assign uniform weights equal to one to all timesteps in the loss, which is also what the original DDPM paper did — and interestingly, it often leads to better results than using time-dependent weighting. We repeat this whole process of noising an image,
27:02
then computing the MSE, until the model is trained, which here takes around 100 epochs. At the end of the training, we save the weights of the network to be able to reuse them later. During training, we can see that the MSE drops rather quickly, even with this very simple setup, and then stabilizes to a very low value. If we display some denoised samples at the end of each epoch, we can track the model’s progress visually.
27:28
Here on the left you can see the original image, then its noisy version at different timesteps, and the denoised estimate computed using the neural network. During the early stages of the training, the outputs are quite poor for high noise levels, but already decent when noise is low. As training continues, the model gets better and better at denoising across all timesteps, but at very high noise levels it
27:52
still struggles to output the correct digit. Actually, it's totally ok for the denoiser to struggle at these high timesteps; it can still perform really well when we use it to generate new samples. Now that training is done, we’ll move on to the inference code and generate new samples from pure Gaussian noise. Let’s now implement the inference code in a separate file. We start by importing our libraries again and instantiating the neural network using
28:22
the pretrained weights we just saved. Next, we re-define all the diffusion constants we used during training — including the linear variance schedule. These parameters must match perfectly between training and inference; otherwise, the amount of noise at a given timestep during inference will not correspond to the same amount of noise seen during training. And now we implement the actual sampling loop.
28:48
The first step is to sample pure Gaussian noise, which corresponds to our initial variable x-t. This noise is then fed into the neural network along with the timestep t to predict epsilon-theta — an estimate of the noise inside this image. This noise estimate can now be used to produce x-t-minus-one, which is a sample from the approximate posterior. To do this we plug the noise estimate into the posterior mean formula and
29:13
add a little bit of noise to actually sample from the approximate posterior. As you can see once again this is exactly the same formula used here in the code. Then we repeat the same process: feed the updated image into the network, compute the posterior mean, and sample the next image — and we typically do this a thousand times. Let's take a look at a few samples, and see how they evolve during these iterations.
29:39
At the beginning of the diffusion process, not much happens — there is still way too much noise to see anything — but when we reach the last few hundred steps, we see the details emerging from the noise. Eventually, we obtain very clean samples, all different from each other. That's pretty incredible right; we’ve just generated high-quality handwritten digits, entirely from scratch, using only a denoiser and a few diffusion equations. What’s even more impressive is that
30:07
the core sampling loop fits in fewer than thirty lines of code. Now here’s the cool part: thanks to DeepInverse, we can easily swap in pretrained weights from different datasets — just by changing one or two lines when instantiating the model. For example, here we load a model trained on F-F-H-Q — a dataset of high-quality portraits. Once again, for the first half of the diffusion, we can't see much within
30:33
all that noise — but once we reach the final iterations, we now see human faces emerging. I don't know about you, but to me these faces are almost indistinguishable from real pictures. And again, the sampling code stayed the same — we’ve just changed the training distribution. Now if we compare these results to those produced by VAEs, the difference in visual quality is pretty clear. Of course, this improvement comes at a cost:
31:00
instead of running the network once, we’re evaluating it a thousand times. So in essence, we’re just trading more computations for a better image quality. What we’d really want, though, is to keep this high visual fidelity — but get it in just a few inference steps. Surprisingly, this was achieved just a few months after the release of the DDPM paper — but that will be the topic of another video.
31:26
Well, it's time to wrap up; in this video, you learned all the basics of DDPMs — how to define a diffusion process, train a denoising network, and generate new samples from pure noise. But there's also a second, more general formulation — one that replaces discrete Markov chains with continuous-time stochastic differential equations. This perspective, known as score-based diffusion, will be the focus of the next video.
31:52
Thanks a lot for watching — if you found this helpful, feel free to like and subscribe to support the channel. See you next month for the next video.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat