Skip to main content
Loading Explore guides…
Probabilistic brain generative models · Acedra
Home
Explore
Computer Science
Probabilistic brain generative models
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
44 segments
Copy
Hide video
0:00
imagine you're walking home one night through a thick fog ahead you see a vague shape moving towards you is it a stray dog a cat or maybe just a garbage bag blown by the wind in this moment your brain doesn't pick a single answer instead it considers multiple possibilities each with its own likelihood this scenario illustrates a fundamental aspect of how our brains process information we don't just see things as binary we interpret probabilities when faced with unclear or
0:37
incomplete information we naturally think in terms of likelihoods and uncertainties this probabilistic view of sensory data also underlies many advancements in modern machine learning in fact a surprising number of computational problems solved by both biological and artificial systems can be viewed through the SLS learning the probability distribution of data performing inference and synthesizing new samples from it a capability often referred to as generative
1:12
modeling the language of probability and uncertainty is built on a few crucial Concepts such as entropy and ke Divergence these ideas are so fundamental that they appear across diverse fields from the latest diffusion models in AI to theories of perception in biological brains like the free energy principle today we'll dive deep into the Core Concepts behind probability starting from the ground up and demistifying many of the formulas that might appear intimidating at first but
1:49
which have a very intuitive understanding if you're interested stay tuned
2:00
before we dive into probability Theory let's address the uncertainty lurking in our online Lives nordvpn who are kindly sponsoring today's video offers an excellent solution to mitigate that digital uncertainty navigating the web is much like walking through that fog we mentioned earlier but instead of mysterious shapes you're surrounded by potential threats to your privacy this risk is particularly High when using public Wi-Fi networks using an unsecured connection is like hoping to have a
2:31
private conversation in a crowded room where anyone could be listening this is where nordvpn proves invaluable it encrypts your data making it unreadable to pry eyes remember those annoying banners that somehow know where you live that's just the tip of the iceberg consider all the other sensitive information potentially at risk passwords personal identity credit card details the list goes on this is why using online protection tools is a must in today's reality nordvpn goes above
3:04
and beyond basic protection with their threat protection Pro feature it not only encrypts your data but also scans downloads for malware blocks suspicious websites and even eliminates those popup ads for anyone prioritizing both security and a seamless browsing experience this feature is a GameChanger if you're ready to take control of your online security visit nordvpn.com / Aram konov to get 4 months extra when you sign up for a 2-year plan still unsure nordvpn offers a 30-day money bag
3:39
guarantee you can try it risk-free so the probability you'll be satisfied approaches one nor VPN is committed to privacy they have a strict no logs policy which means they don't track or store online activities it is also the fastest VPN on the market ensuring you won't even notice it's running don't leave your online security to chance head to nordvpn.com sardam konov link in the description to claim the exclusive offer the most fundamental Concept in our today's discussion is the notion of
4:13
a probability distribution in general we deal with an abstract set of possible States and to each of those States we assign a positive number between zero and one this number reflects our degree of belief in How likely that particular state is to occur you're probably familiar with thinking about probability in terms of the proportion of outcomes from high school for example a six-sided die has six possible states to find the probability of rolling a three for example you would roll the die many
4:52
times count how often it lands on three and divide it by the total number of rolls as you increase the number of rolls this ratio converges to 1/ 6 which is the probability for any site of a fair die this approach which relies on repeated trials and tracking outcomes is called the frequentist interpretation of probability while this interpretation works well for dice or coin flips it has limitations consider a weather forecast stating the probability of rain tomorrow
5:32
is 70% we intuitively understand this but it doesn't fit neatly into the frequen definition we can't repeat tomorrow's weather multiple times to find the frequency of rain outcomes tomorrow will happen only once and it will either rain or not this is why we'll adopt the basan view which treats probability more generally as the degree of belief when we say rain with probability 0.7 it means we've assigned a 0.7 degree of certainty to it while this definition might seem very broad it still obeys
6:14
certain laws the most crucial one is that all the degrees of belief you assign to possible states must sum to one for instance if there are only two outcomes rain and sun then saying the probability of rain is 0.7 implies that the probability of sun is 0.3 this brings us to the notion of a probability distribution essentially it is a function that Maps each possible state to its assigned probability value because probabilities must sum to one we can't manipulate this function for one
6:53
state independently of others it characterizes the system as a whole completely describing how uncertainty is distributed among individual states here are a few examples of probability distributions for a fair die the distribution is uniform each side has a possibility of 1 / six the distribution of heights of adults in a population follows the famous bell-shaped Goran distribution there is a technical detail here since height is a continuous variable the probability distrib
7:29
distribution must integrate to one meaning that the area under the entire curve equals to one but this is just a general case of summation you can think of continuous distributions as being made up of a large number of tiny discrete bins it won't change the Core Concepts we are discussing so far we've only looked at the probability distributions over states that can be neatly arranged on a one-dimensional line but what about more comp scenarios like states that include both
8:03
height and weight we could try to put them on a one-dimensional line but that's not very enlightening instead it is more natural to think of States as occupying a two dimensional plane with one coordinate for the height and one for the weight then each point on this plane corresponds to a particular State and we can represent its probability with color or as a third coordinate for more coordinates we can't directly visualize higher Dimensions but the idea Remains the Same in machine
8:40
learning we often deal with much higher dimensional probability distributions for instance all possible 100 by 100 pixel images occupy a 10,000 dimensional space with one axis for each pixel's value we can then reason about the probability to distribution of images by assigning a probability to each point in this space but how do we actually use or interact with these complex distributions to understand this let's go back to our simpler example and think about the reverse process of generating
9:18
outcomes from a probability distribution this is called sampling and it is a crucial concept that is at the heart of many generative AI models in the case of a die rolling it generates a new sample from the underlying distribution picking one state at random with a certain probability similarly if we have a way to draw samples from a 10,000 dimensional distribution we can generate new images because this distribution captures the structure of data present in natural
9:54
images this would be entirely different from randomly selected values for each pixel independently as this would always result in noise now that we have established a common ground for the language of probability let's explore the powerful math behind what it takes to describe and model a distribution let's start with the notion of surprise imagine that before you toss a coin someone comes up to you and correctly predicts the outcome will you be surprised to see see
10:29
this prediction hold well maybe but not too much after all the chance of guessing was 50/50 now suppose you do the same for rolling a die and someone correctly predicts that it would land on one you'd likely be more surprised by this rather than a coin right let's try to formalize this notion of surprise essentially we're looking for a way to assign a positive number a surprisal measure to each state this measure should be a function of probability that gets larger for smaller
11:08
values and shrinks to zero when the probability approaches one after all events that are guaranteed to happen should have no surprise at all a crucial but less obvious aspect is the additive nature of the surprise if someone correctly predicts the outcomes of three dice rolled simultaneously would be about three times as surprised as for one die but the probability of such an event would be the product of individual die probabilities in other words when probabilities multiply as for
11:47
independent events the surprisal function should add the function that behaves this way is the logarithm of one over the probability the this makes intuitive sense surprise is high for rare events and low for common outcomes now that each state has its own surprise we can characterize the entire distribution by finding the average surprise packed into it this average surprise is what we call the entropy of that distribution mathematically it is expressed in the following way where we
12:29
sum over all possible States multiplying the probability of each state by its surprisal
12:40
value for example consider a thick coin that lands on heads and tails with probability 49% each and on its side 2% of the time if you randomly flip it many times record your surprisal caused by the outcome of each flip and compute the average surprisal what value are you going to get well every time you get either a heads or a Tails which in total comprises 98% of all the flips your surprise is about 0.7 but in the remaining 2% your surprise will be around four so the
13:23
weighted average given by this formula is approximately equal to 0.8 compare this to the entropy of a fair coin that always lands on heads or tails this value is less which means you will be on average more surprised by observing outcomes of a coin that lands on its side in other words it has more inherent uncertainty packed into it now I'd like to highlight an important Point Computing entropy relies on no the probability distribution but in real world when we
14:03
observe a random variable unfold we don't have access to the true underlying probability distribution instead we usually have an internal model what we believe the probability distribution should be and this is what we use to assign the probabilities the true underlying distribution is hidden and typically far too complex so all we have are approximate models of it for instance consider a fair diey the true probability distribution of its sides slightly deviates from uniform due to
14:42
inherent microscopic imperfections in the diet s air resistance Etc it may even include other outcomes such as Landing exactly on a corner or with vanishingly small probability falling apart in the air however we approximate this with a simple uniform distribution 1/ 6 for each side this model captures everything we need for practical purposes but what if the deviation between our internal model and the actual distribution is more drastic let's consider an example
15:23
suppose someone gives you a coin to flip based on your prior experience with coin you believe the probabilities for heads and tails to be around 50% each what you don't know though is that this particular coin is rigged with its Center of mass shifted so that it lands on heads around 99% of the time you go ahead and flip it 10 times getting 10 heads in a row under your current model of a fair coin the probability of this happening is extremely small so you will be very
16:02
surprised however the actual true probability of this outcome is quite High because the coin is rigged notice that the surprisal you experienced is caused by your belief about the probability distribution being wrong not because something truly remarkable had happened this brings us to the idea of cross entropy essentially it is a function of two probability distributions that quantifies the average surprise you might expect when observing a random process generated by
16:38
one probability distribution while believing it comes from another mathematically the cross entropy between distributions P the true one and Q your model is defined similarly to entropy but we use P for the outcome probabilities and Q for the surprisal terms the surprise given by cross entropy can come from two sources the discrepancy between your model and the true distribution as well as the inherent uncertainty of the underlying distribution itself if p and Q are the
17:18
same distribution meaning that you have a perfect model the cross entropy is simply equal to the entropy of p a key mathematical property which we're not going to prove rigorously here is that the cross entropy of distribution p as the ground truth and Q as its model is always greater than or equal to the entropy of P itself in other words believing in their own model of a random variable can only increase the surprise you'll get by observing it never decrease it another important feature of
17:57
cross entropy is its asymmetry the roles of p and Q matter and swapping them can lead to very different results let's explore this with two examples first believing a coin is fair when it is actually rigged suppose the true distribution p is a rigged coin 99% for heads 1% for Tails but your model Q is a Fair coin for a sing single flip the cross entropy can be calculated in the following way and is equal to approximately 0.7 now let's reverse the scenario what
18:40
if you believe a coin is rigged when it is actually Fair Computing the cross entropy gives us 2.3 this value is much larger than the other way around this is because half the time when you see tails you will be extreme ly surprised as your model predicted this was very unlikely the other half of the time when you see heads you'll have a very low surprise but the extreme surprise from Tails dominates leading to a high average surprise measured by the cross entropy our examples of crossentropy
19:21
highlight an important question how can we quantify the difference between our model's beliefs and the reality can we somehow isolate the part of the surprise that comes purely from our models inaccuracy rather than from the inherent uncertainty in the data this is where the concept of callback lier Divergence or key Divergence for short comes into play it allows us to peel away layers of surprise recall our definition of cross entropy this measures the total surprise
19:57
when using Q to predict ICT P now what if we subtracted the entropy of P from this this formula represents the inherent uncertainty in P Itself by subtracting it from the cross entropy and using the properties of logarithms we're left with this this is the keyl Divergence it measures the extra surprise we get from using Q as our model instead of P beyond the surprise inherent in p ke in our coin analogy it is like isolating the surprise caused by believing in the wrong model separate
20:37
from the surprise caused by the coin flip itself but why might we need to measure this discrepancy in the first place well the ultimate goal of building and training many machine Learning Systems is to construct a good approximation a model of the underlying probability distribution of some data to later use it for prediction or sampling for example suppose you're building an AI system that can create new lifelike photos of cats ideally you'd want to sample directly from the distribution of
21:14
all possible cat images in the real world however that is an incredibly complex space that we can't directly access or describe so we'll have to resort to building a model to approximate it without diving too deep into the specifics of generative models in this video I will just say that we usually approximate the shape of that distribution with neural networks these can be viewed as mathematical functions with enough expressive power to capture complex patterns once we have parameterized the
21:51
distribution with a neural net of some kind we can optimize its parameters with techniques like gradient Des if you're not familiar with gradient descent or back propagation you can check out my previous video on the topic but how do we know whether a particular approximation is a good or a bad fit to the true Target distribution after all we need some kind of error function that we'll be trying to minimize you might have guessed that the KE Divergence between our model and
22:23
the true distribution is exactly the target objective we would like to make make as low as possible ideally we'd aim for zero which would mean our model perfectly aligns with the data distribution however in practice if you look at the code for training models to capture a probability distribution you typically won't see the key Divergence as a loss function instead you're likely to see the optimization objective framed as minimizing the cross entropy so what's the catch here let's look at the
22:57
definition of ke Divergence once again we're trying to find the model q that would minimize this quantity however notice that the term h of P does not depend on our model at all it is a constant determined by the true distribution P which we can't change because it stems from the inherent uncertainty in the training data that is fixed no matter how we tweak the parameters of a neural network it has absolutely no effect on how diverse cats in the real world are this is why
23:32
whatever approximation Q minimizes the KE Divergence it also by definition minimizes the cross entropy in other words the two objectives are equivalent the entropy of data term does not affect which Q is optimal it just shifts the value of K Divergence by a constant amount Computing the exact value of ke Divergence would require estimating the entropy of TR training data from a finite number of samples although you can certainly do it usually we don't really care about what the actual value
24:06
of the Divergence is as long as it's the lowest one possible it doesn't affect the resulting model so we just don't bother with estimating the entropy of data in the first place to save some computation this forms the basis for many machine learning algorithms especially in the realm of generative models by Framing our objective in terms of entropy minimization we can train our models to approximate complex real world distributions even when we don't have direct access to those distributions
24:39
while minimizing the cross entropy is at the very core of mod than generative models training complex systems like image generation networks in practice often requires additional techniques that go beyond the scope of this video These may include things like variational which we might explore in the future my main goal today was to lay a solid foundation for understanding what it means to learn the probability distribution of data we have explored fundamental concepts like entropy
25:13
crossentropy and ke Divergence showing how they are rooted in intuitive ideas of surprise and uncertainty these principles are not just abstract mathematical constructs they form the bad bone of how we quantify model and learn from data in the real world so hopefully next time you encounter terms like crossentropy in the minimization problem you will recognize it not as a mysterious formula but as an intuitive idea of measuring and minimizing the gap between our
25:48
models and reality if you like the video share it with your friends press the like button and subscribe to the channel if you haven't already stay tuned for more computational neuroscience and machine learning topics coming up goodbye and thank you for the interest in the brain
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat