Skip to main content
Loading Explore guides…
Rise of Bayesian Statistics · Acedra
Home
Explore
Mathematics
Rise of Bayesian Statistics
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
38 segments
Copy
Hide video
0:00
According to the database at ScienceDirect.com the number of research publications mentioning the term "Bayesian" has been increasing every year since the year 2000. If we look at the breakdown by year, we can see that there was a noticeable change in in the trajectory around the years 2017 and 2018. The underlying concepts behind Bayesian Statistics have been around since the 1700s. So, what's fueling this recent interest, and why should you consider learning it? Well, let's find out!
0:36
[golf ball hit sound] Welcome to doggo.jl, where I explore the vast Julia wilderness and turn my discoveries into wholesome Julia tutorials. [golf ball hit sound]
0:50
Ben Lambert is a researcher at the University of Oxford and is the author of a book titled "A Student's Guide to Bayesian Statistics", which was published in 2018. He also has a YouTube channel that contains a playlist of videos that supplements his book. This playlist description provides an excellent motivation for why you should consider learning the subject. His articulation is much better than anything I could come up with, so I'm just going to quote him directly:
1:24
"There are some ideas we know are true," "and others we know are false." "However, for the majority of ideas," "we do not know if they are true or false;" "for these we say, we are 'uncertain'." For these uncertain entities," "the correct way to quantify our ignorance," "is to use the language of probability." "Bayesian inference uses probability theory" "to allow us to update our uncertain beliefs" "in light of data." "Bayesian inference is increasingly used" "across the social,
2:03
"biological and physical sciences." "As such," "a working knowledge of Bayesian Statistics" "is essential for researchers in these fields." [golf ball hit sound]
2:17
When speaking casually, the term "probability" refers to some number between 0 and 1 inclusive, which is often stated as a percentage between 0% and 100%. So, when we hear a weather forecast with a 70% chance of rain tomorrow, we understand that it's more likely to rain than not to rain at some point tomorrow. But mathematically, how is probability interpreted? Historically, probability has been interpreted either objectively or subjectively. The Objectivist interprets probability
2:58
as a measure of the real physical tendency of something to occur. In contrast, the Subjectivist interprets probability as a measure of how strongly one "believes" it will occur. For the majority of modern history, the Objectivist point of view was the dominant view in the field of mathematics. But in recent years, the Subjectivist point of view has been gaining in popularity. Perhaps the best known application of Subjective Probability is Bayesian Statistics. [golf ball hit sound]
3:38
Despite it being named after Thomas Bayes, who is known for Bayes' Theorem, Bayesian Statistics was actually developed by the French scholar and polymath, Pierre-Simon Laplace in the late 18th and early 19th centuries. What distinguishes Bayesian Statistics from traditional statistics is that the Bayesian interpretation of probability is that probability expresses a subjective "degree of belief" in an event. Bayesian Statistics starts with a Prior belief about a probability
4:15
and then uses Bayes' Theorem to update that belief based on new information. [golf ball hit sound]
4:24
Thomas Bayes was an English statistician, philosopher and Presbyterian minister who, in the 1700s, formulated what would come to be known as Bayes' Theorem. Bayes' Theorem describes the probability of an event based on prior knowledge of conditions that might be related to the event. Mathematically, Bayes' Theorem is stated using this equation, where A and B are events, and and the probability of event B occurring is not equal to zero. The vertical bar means "given",
5:03
which is the mathematical notation used for a Conditional Probability. So this equation is read, "The probability of event A occurring," "given that B is true," "is equal to" "the probability of event B occurring," "given that A is true," "times" "the probability of observing event A," "divided by" "the probability of observing event B." This equation may look simple, but it's been interpreted and applied in several different ways throughout the years. One of the ways that it's been used
5:42
is in Bayesian Statistics, for something called Bayesian Inference. [golf ball hit sound]
5:50
In statistics, the term "inference" refers to the practice of drawing samples from a population. Based on the characteristics of those limited number of samples, one can "infer" the characteristics of the larger population. Bayesian Inference uses Bayes' Theorem to update a Prior belief about a probability as more information becomes available. So, how does this work? In Bayes' Theorem, the probability of A, given B, is called the "Posterior Probability". The probability of B, given A,
6:30
is called the "Likelihood". The probability of A is called the "Prior Probability". And the probability of B is called the "Marginal Probability". The Marginal Probability is often difficult, or sometimes impossible to calculate. But its role is essentially to normalize the probability distribution, so it's often left out, leaving this equation that states that the Posterior is equal to the Likelihood time the Prior. The Turing.jl package requires you to define the Prior before the Likelihood.
7:13
So, when using Turing, you might see this equation stated as the Posterior is equal to the Prior times the Likelihood. So, how can we interpret these terms? In Bayesian Inference, event A is replaced by the parameter, or parameters, of interest. And event B is replaced by your observation, or observations. As a way to shorten the expression, the parameter, or parameters, is denoted by the lowercase Greek letter, 'theta'. And your observation, or observations,
7:52
is denoted by the uppercase letter, 'X'. So you may see the equation written as P of theta given X is equal to P of theta times P of X given theta. But regardless of how it's written, they all mean the same thing. In English, this equation means that you start with some Prior belief, about something, before making any observations. And then, you either witness some observations, or collect some sample data. Based on this new information, you can use this application of Bayes' Theorem
8:32
to update your belief. "Wow!" This may sound simple, but the reason why Bayesian Statistics and Bayesian Inference were not widely used until recently, is because the math required to solve Bayes' Theorem is often too difficult, or sometimes impossible, to calculate by hand. But with the advent of modern computers and modern programming languages, Bayesian Statistics and Bayesian Inference are now gaining wider acceptance. [golf ball hit sound]
9:05
The use of modern computers and modern programming languages for Bayesian Statistics is called Probabilistic Programming. Probabilistic programming languages have been around in various forms for decades. The release of the Stan programming language in the year 2012 marks an important milestone in the recent proliferation of Bayesian Statistics. Stan is an open-source, domain-specific language for Probabilistic Programming written in C++. It was created by a team of developers,
9:41
many of whom have a connection to Columbia University. Since it was a pioneer in this field, it's no coincidence that Richard McElreath and Ben Lambert both include Stan code in their books. But since its initial release, there have been several other Probabilistic Programming libraries, or packages, written in other programming languages. One such package, written using the Julia programming language, is Turing.jl. Turing.jl was created by Hong Ge at the University of Cambridge,
10:18
and it was originally released in the year 2018. Unlike Stan, which is a domain-specific language, Turing is a Julia package, so it can be used like any other Julia package. By using Turing, there's no need for you to learn another language. But regardless of which language, or package, you decide to use, the workflow for Probabilistic Programming is similar. [golf ball hit sound]
10:48
In this section, I'll be referring back to the Julia code from the last tutorial, Episode 901. When using Turing.jl, there are two major parts. The first part is provided by the user, and the second part is automated thanks to Turing. These are the pieces that the User provides. 1. Observe an Event, or Events. 2. Define a Model. And 3. Select an Inference Algorithm. The automated section only has one step. 1. Use the Inference Algorithm to build a Chain, or Chains. And that's it!
11:35
This Chain, or Chains, is the approximation of the Posterior Probability. Conceptually what's going on is that you start with some hypothesis before making any observations. As a Bayesian, you believe that there exists some parameter that, when fed into a probability model, will generate Events. Before witnessing any Events, you hold some Prior belief about the probability distribution about the value of that parameter. And then, you make your observations,
12:11
either by designing a controlled lab experiment, or by going out into the field to collect sample data. After you go through your sample data, you can then feed that new information into a Bayesian probability model in order to update your belief about the probability distribution surrounding the value of the input parameter. This is an iterative process, so you can continue to update your belief as you're presented with new information. But going back to the Globe Tossing Experiment
12:45
from the last episode, we were simply given the observation data of 9 globe tosses, resulting in 6 of the 9 tosses being Water, as opposed to Land. When we create our Model using the @model macro this is the application of Bayes' Theorem in the form of the Posterior equals the Prior times the Likelihood. The input parameter that we want to understand is the variable named "percent_water". Our Prior belief, before seeing any data, is that this "percent_water" parameter
13:22
is distributed as a uniform Beta distribution. The Likelihood is the Probability of the Observations given the Parameter. In other words, if you knew the value of the input parameter, how likely is it that this input parameter would generate your observations? In this example, our observations are captured in the variable named "water". And we believe that is likely that our observations are distributed as a Binomial distribution, based on the number of tosses
13:56
and the value of the input paramter named "percent_water". This model will be used to update our belief about the probability distribution of the input parameter. This updated belief, after seeing the data, is called the Posterior Probability. [crowd gasping] But in many cases, we cannot calculate this Posterior Probability directly. Instead, we need to use a random sampling process in order to infer the distribution of the input parameter. By using Bayesian inference,
14:38
we can find an approximation of the Posterior Distribution of the input parameter. In Turing, taking samples is accomplished by using the sample() function. The sample() function takes 3 arguments: a model, a sampler, and the number of samples. The sampler is an algorithm that defines how your computer takes samples from the model. A widely used class of inference algorithms used for Probabilistic Programming is known as Markov Chain Monte Carlo, or MCMC. [golf ball hit sound]
15:18
When using Bayesian Inference, you can use many different random sampling processes. But as your models get more complex, many random sampling processes become computationally expensive, so there's a need to use more intelligent sampling processes. Markov Chain Monte Carlo, or MCMC, is the name given to a class of algorithms for randomly sampling from a probability distribution, but in a way that is more intelligent and more efficient than a naive random sampling process.
15:55
There are several algorithms that fall under the Markov Chain Monte Carlo umbrella. The MCMC algorithms that you can use with the Turing.jl package include Gibbs, Hamiltonian Monte Carlo, Hamiltonian Monte Carlo with Dual Averaging, Importance Sampling, Metropolis-Hastings, No-U-Turn Sampler Particle Gibbs, and Sequential Monte Carlo. The sampler that we used in the Globe Tossing Experiment was the No-U-turn sampler, which is considered a state-of-the-art MCMC sampling algorithm.
16:36
But as you can see, there are several different algorithms that you can use, and Turing makes it very easy for you to try out different samplers. Please refer to the Turing.jl documentation to learn more about each sampler. So getting back to our workflow, once you select a sampler, you can then use the sampler to try to figure out the shape of the Posterior Probability of the input parameter. At this point, we don't know what that shape looks like but we have an idea
17:07
of the proportionality of that distribution based on our beliefs about the Prior and the Likelihood. So the sampler selects random values for the input parameter in the frequency of the probability distribution based on those beliefs about the Prior and the Likelihood. The sampler will examine these samples in an attempt to infer the shape of the Posterior Probability. The reason why the MCMC algorithms are more intelligent and efficient compared to a naive random sampler
17:42
is because the MCMC algorithms are built with selection criteria that seek samples fitting the proportionality of the Posterior Probability. These algorithms will also discard the samples that don't fit their selection criteria. These algorithms will continue selecting random samples based on the number of samples that you set. The result of this inference process is called a Chain. This Chain is the approximation of the Posterior Probability. [golf ball hit sound]
18:18
The Posterior Probability is your updated belief after making your observations. So, going back to our Globe Tossing Experiment, our Prior belief about the probability distribution of the input parameter named percent_water was that the value was uniformly distributed. This meant that we believed that the value of the input parameter could have been any value between 0 and 1 with equal probability. But after tossing the globe 9 times, we observed that the result
18:53
was that our finger pointed to water 6 out of the 9 times. Based on this new information, we were able to infer that the shape of the probability distribution of the input parameter looked more like this. So our updated belief is that the probability distribution of the input parameter is weighed it heavier, somewhere between 60% and 80%. Since this is an iterative process, we reserve the right to update our belief again if we're presented with new information. [golf ball hit sound]
19:31
So now that we know some of the key concepts behind Bayesian Statistics and Probabilistic Programming let's revisit Ben Lambert's motivating quote. "There are some ideas we know are true" "and others we know are false." "However, for the majority of ideas," "we do not know if they are true or false." "for these, we say we are 'uncertain'." "For these uncertain entities," "the correct way to quantify our ignorance" "is to use the language of probability."
20:05
"Bayesian inference uses probability theory" "to allow us to update our uncertain beliefs" "in light of data." You can see why these concepts may resonate within the scientific community. Scientific research is all about exploring the unknown and Bayesian Statistics and Probabilistic Programming provide the researcher with a set of practical tools to help navigate in a world full of uncertainty. [golf ball hit sound]
20:38
If you made it this far, Congratulations! [kids cheering] If you enjoyed this video, and you feel like you learned something new, then please consider Liking Commenting Sharing and Subscribing. If you'd like to support the educational work that I'm doing, then please consider using the Super Thanks button. For ongoing support, please consider Joining and becoming a Channel Member. Channel Members get ad-free, early-access to all of my new videos. Thanks for watching and good luck
21:10
on your Julia journey!
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat