Skip to main content
Loading Explore guides…
Moment Generating Functions · Acedra
Home
Explore
Mathematics
Moment Generating Functions
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
37 segments
Copy
Hide video
0:00
welcome back so I'm really excited today to introduce you to one of my favorite Concepts in probability the moment generating function so I remember when I was learning probability you know ages ago this was one of the kind of biggest aha moments for me that connected a lot of different topics so the moment generating function is roughly speaking going to be a nice function that helps us compute moments things like the expectation the variance and higher order moments things like skew and
0:32
curtosis in a pretty nice compact formula and it's also going to be Central in proving some of the most important theorems in probability and statistics like the central limit theorem okay so what is a moment in probability the the moment of a distribution the moments of a probability distribution are the expected values of my random variable X or X2 or X to the^ n so there are actually infinitely many moments there's a first moment a second moment a third moment fourth moment and so on and so
1:07
forth and they are the expect expectation value of my random variable to the power n so we've already seen examples where this is super important the mean is the first moment the mean of my distribution is the first moment and the variance of my distribution is closely related to the second moment it's the second moment minus the first moment squared and these are extremely important for characterizing my probability distribution function so if I know that my distribution is let's say a normal
1:41
distributed random variable X or a gausian then it is uniquely determined by the first and second moments so the first and second moments uh uniquely determine kind of the mean and the standard deviation this is let's say mu plus Sigma and minus Sigma the um the mean and the standard deviation of that uh normally distributed random variable but higher order moments are actually also important there are lots of distributions where you need more moments to uniquely determine that
2:16
distribution so the third moment uh the expected value of x to the^ 3 is known as the skewness of the distribution so if I have a distribution that's a little bit lopsided something like this it's going to have some skewness so the pan distribution for example is going to have some nonzero skewness um if I have a distribution that has some you know kind of fat tails I might need my fourth moment or my curtosis the expected value of x to the 4th um so I can draw you know a picture
2:46
of that maybe I have something that looks kind of like my normal distribution but I've got some bumps out here in uh in the Tails there would be some fourth moment or some curtosis in that distribution and these moments are unique identifiers of my probability distribution it's a lot like the derivatives of my function in the tailor series approximation so if I want to approximate a function with a tailor series I can you know collect all of the derivatives the nth derivatives
3:16
of my function and uniquely write the tailor series for analytic functions same basic idea applies for probability distributions you can uniquely determine uh the the let's say the cumulative distribution function and sometimes the probability distribution function in terms of this kind of infinite series of moments so this is kind of like the tailor series for probability and statistics super super fundamental concept and very very useful okay so let's define this moment generating
3:45
function which is going to allow us to compute these moments very efficiently okay um so I want to do this in pink I think because I like my uh probabilities to be here so the moment generating function uh is going to be defined in terms of an entirely new variable usually these are defined in terms of a variable X here we're def we're introducing a new variable because we're going to do some transformation uh from probability space into this moment generating space and this is defined as
4:15
I'm going to write it for a discret variable X and for a continuous random variable X um so for a discret random variable X it's defined as the sum over all possible States X can take of e to the T x times the probability density function P that big x equals Little X um this is for X discret for a discret random variable okay this is just how it's defined I'll tell you why it's defined this way and how to use it in a minute and for continuous random variables for x uh continuous things
4:53
like the normal distribution or an exponential distribution or a gamma distribution this is defined as the integral over all possible X's so you know let's say generically from negative Infinity to Infinity of e to the TX times my probability density function f ofx DX now first off those of you who have studied differential equations um control theory dynamical systems you'll recognize right off the bat that this is the llao trans form of my PDF this is the lus uh
5:35
transform of my PDF my probability density function f ofx which is pretty wild that this very useful function for approximating these moments happens to be the the Lelo transform of my probability density this actually maybe shouldn't be so surprising Lelo was one of the absolute founders of modern probability and statistics in fact beian statistics I think should be really called beian llas statistics because even though it was discovered by Baye a couple of years earlier llas
6:08
independently discovered it and went way farther in developing it so most of our modern theories and kind of language and thinking about measurement error probabilities and statistics are actually responsible um because of of llas so llas must have been thinking about this transform both in terms of probabilities and differential equation and there's some deep deep connections if you think about stochastic processes like um you know radioactive decay you have a differential equation and a
6:38
corresponding probability density function and you can take the LL transform of both and it means something so lot of deep connections but zooming out this moment generating function is something you can compute it's the Lao transform of your probability density function and it has this very very useful property that the moment generating function can easily help you generate the moments the higher order moments of your probability of your your random variable X so the way that looks
7:12
um and maybe I'll do this in in blue this is a theorem I'm going to prove next time in the next video but I'm going to State it here so there's a theorem that the moment generating function the nth derivative of my moment generating function evaluate at zero is equal to this expectation of x^ n this is super useful so if I take the first derivative of this function and evaluate it at zero I get my expectation my my mean if I take the second derivative of this function and evaluated at Zer I get
7:48
the expectation of x^ squ the third derivative evaluated at zero gives me something related to my skewness fourth derated derivative related to curtosis so all of my moments I can I can generate I can easily kind of grab from this moment generating function super super useful because these are quite difficult to calculate generically um and once I have this function it might be much much easier to do this and again this is going to uniquely characterize maybe I'll write this down this moment
8:17
generating function M oft uniquely determines uh determines the cumulative probability distribution the cumulative probability distribution specifically the probability that my random variable X is less than or equal to some little value X okay and this is something we'll we'll see later now notice that I didn't say the probability density the the PDF I said the cumulative distribution function the CDF there is a reason why I talk about cdfs often in terms instead
8:58
of PDFs I'll tell you later but just gloss over this fact and just say this moment generating function uniquely determines the distribution your probability distribution okay and it's very very useful for computing these moments that are again kind of the the the fingerprint or the the tailor series um kind of approximation of your distribution these uniquely determine your distribution great so I want to give you some examples how do you do this for Pon how do you do this for
9:28
normal how do you do this for x itial and next time I'll prove this theorem uh that my moments uh give me these that my moments can be drawn From This Moment generating function okay good um so I think from now on I'm just going to be doing some examples um is there anything else I want to tell you I think that's pretty good moments are important they uniquely determine your distribution and you can pull them out of this kind of Magic Moment generating function which
9:55
happens to be the laa transform of your PDF pretty profound pretty powerful stuff let's do some examples um so I'm going to do pan normal and exponential so example let's start with Pon um the pon distribution so my PDF here this is a discret random variable so let's say x is Pan um we'll say you know X is pan with some Lambda and of course that means its PDF um is going to B what's its PDF it's Lambda to the K eus Lambda over K factorial that's the PDF of Pon
10:40
so the moment generating function is going to be the sum of e to the T K sum over K this is my moment generating function of T okay so here it's defined in terms of some dummy variable X here my dummy variable is K because that's what we normally do for Plus on but it's this expression here and it's not that hard to manipulate this and get a pretty easy expression so my moment generating function is this which is the sum I'm going to pull this e to the Lambda out
11:13
of my sum because the sum is over K this is e to the minus Lambda sum of e to the TK Lambda to the K that's e to the T Lambda to the power k divid K factorial I pulled out my e to the minus Lambda and then I combined these terms into e to the T Lambda all of that to the power K ID K factorial pretty simple stuff now this expression here the sum of this thing to the K Over K factorial this is the definition of e to the stuff inside the parentheses so this equals e to the minus Lambda time e
11:55
to all of the stuff in the parentheses e to the goodness Lambda e to the T all this sum here over K this is the definition of the exponential of all the stuff in this parenthesis so it's e to the Lambda e to the T * eus Lambda so this equals e to the Lambda did I do did I have a minus sign here um yeah e to the Lambda e to the T minus one okay so this is the moment generating function M of t for the pon distribution and I can take its first derivative and evaluate at zero to get
12:40
the mean I can take its second derivative evaluate at zero to get something related to the variance and so on and so forth this is an easy function I can work with this is how you compute the moment generating function of a discrete random variable like Pon okay let's try another one let's do a normal distribution okay example two normal distribution so let's say that X is in normal and let's make it really easy on ourselves let's say it's mean zero standard deviation one I don't want to
13:13
deal with you know some hairy normal distribution just a basic standard normal um okay we're going to write this out we're just going to do it it's not going to be that bad and it's going to save us time later when we want to compute higher order moments and and do things like that this is also really important for the central limit theorem so the moment generating function uh of this um PDF is the integral from negative Infinity to Infinity of e to the TX time my PDF DX um so it's
13:48
integral negative Infinity to Infinity of e to the TX and my PDF of my normal has a 1 over root 2 pi I'm going to pull that out because it's just a normalization factor 1/ < tk2 Pi time e to- x^2 over Sigma squ which is just uh - x^2 over yeah over 2 Sigma squ and since Sigma is 1 my PDF is just e- x^2 over 2 okay so I pulled my normalization factor out for my my gausian normal distribution and this is the PDF of my normal distribution and this is my plus transform integrating Factor so this is
14:31
the thing I need to compute and pretty easy I'm literally just going to combine these exponentials do some really basic algebra and integrate this thing okay so one tiny fact that's going to help us do this is that x^2 / 2 minus TX that's just the minus of these exponents when I multiply this this thing is equal to uh 12 x^2 - 2tx which is equal to x - t^ 2ar - t^ 2 really really simple if you take x - t ^ 2 um you know times a half and then you subtract off the t^2 term you're just
15:25
left with an X2 and a 2tx okay this is really really simple just you know um expand your your polinomial whatever this is just an identity to take this exponent here when I multiply these two and write it in terms of these squared exponents that are going to be a little bit easier to work with this is just an identity and so now this integral here is equal to 1 over < tk2 Pi integral minus infinity to Infinity of um of let's see this thing here which is e to the min-2 uh x - t^ 2ar
16:12
plus 12 t^ 2 all of this integrated with respect to X so this is um TX - x^2 over 2 which is the minus of this identity so I get the minus of this identity which is 12x - t ^2 Plus + 12T ^2 we're just doing integration now and now I can split these integrals up uh I can split this this product up here so this equals my e to the 12 t^2 doesn't depend on X so I can pull that outside of my integral so now I get one I get e to the 12 t^2 * 1/ < TK 2 pi uh 1 over < tk2 Pi
16:56
integral minus infinity to Infinity of e to -2 x - t^ 2 DX now this is the probability distribution for another normally distributed variable with mean t and standard deviation or variance equal to one so this thing also has integral probability equal to to one if I integrate this from negative Infinity to Infinity the law of total probability says this integral equals one and so my moment generating function is just this stuff on the outside it's just equal to e to the t^2 / 2 This Is My Moment
17:37
generating function for a normal pretty simple stuff okay I mean like there was a little hairy math in the middle but I think you followed this is calculus it's easy we know how to do this and now we have this really simple function for the moment generating function this is actually a nicer function than my PDF in a way it's it's closely related but it doesn't have all my normalization stuff okay let's do one more example before we close out um so the last example I'll do
18:05
is my exponential distribution okay so now I'm going to do example three uh exponential and my exponentially distributed random variable is pretty closely related to radioactive decay and things like that so you know that's where your Odes and your Lao transforms are going to come in if you really wanted to follow up on this so the moment generated function is again equal to the integral in this case from 0 to Infinity because my exponential distribution only is for positive times
18:37
um and here I'm going to introduce a different dummy variable s for my Lelo transform because my exponential already is defined with a variable T and it's going to get confusing so we're going to say this is e to the S times my probability distribution an exponential which is Lambda eus Lambda t T and now I'm integrating this with respect to t uh and now my moment generating function is actually in terms of a dummy variable s um this is actually more like a Lao
19:11
transform because my exponential we're already using the variable T I'm integrating uh you know out DT and my llao variable now is going to be S and I'm going to get a moment generating function in terms of s totally fine you can call these variables anything you want um and this thing is really easy to compute so I can pop my Lambda out and I get um what do I get I get Lambda integral 0 to Infinity of e to S minus Lambda * T DT and this is really easy to compute
19:48
this integral this is just how you do llao transforms in general this is Lambda over s minus Lambda e to S minus Lambda T evaluated at the bounds of integration infinity and zero so this is only defined if s is less than Lambda okay if s is bigger than Lambda this thing is going to blow up we're assuming real valued s for now and so s has to be you know less than Lambda for this to to work so this is defined uh if s is less than Lambda and if s is less than Lambda then the
20:30
infinity bound this goes to zero and the zero bound this goes to minus1 so this equals Lambda over it's times minus one so it's Lambda over Lambda minus s this is the moment generating function for an exponential distribution and it's only defined for some values s um depending on Lambda okay so three different examples of how to compute this moment generating function it's pretty easy for discret VAR variables for continuous variables the functions aren't too nasty and you
21:02
can use these to to calculate the moments the expectation the variance the skewness the curtosis the fingerprint of your distribution for really really you know a wide variety of of probability distributions and again last thing this will be useful when we start to prove the central limit theorem that if I add up n independent but identical random variables if I add up a bunch of the same distribution random variables they will start to converge to a normal distribution we're going to use the
21:37
moment generating function to prove that we're going to show that that sum that the moment generating function of that sum of of random variables starts to converge to the moment generating function of the normal distribution so we're going to use this exact result to prove the central limit theorem okay thank you
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat