Skip to main content
Loading Explore guides…
Hypothesis Testing · Acedra
Home
Explore
Mathematics
Hypothesis Testing
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
50 segments
Copy
Hide video
0:00
Give me 20 minutes and I promise you hypothesis testing will click forever. You're about to master the universal algorithm. The one algorithm that powers every AB test, every clinical trial, and every quantitative trading strategy. First, let's learn the language. It's actually pretty simple. First up is Hnot, the null hypothesis. This is just the boring status quo, the default reality. Then you have H1, the alternative. This is the exciting new theory you want to prove. Next
0:24
is alpha. Think of this as your BS threshold. It's the maximum risk you're willing to take of being wrong. and we usually set it at 0.05. And finally, the p value. This is the probability of seeing your data if that boring null hypothesis is actually true. Here's the secret. It's just four steps. So, what is this universal algorithm? It's a function. It takes your data and it spits out a decision. That's it. Step one, formulate. You define Hnot, your boring theory, versus H1, your
0:48
exciting theory. Step two, calibrate. You set your alpha, like 0.05. This is where you ask yourself, how much evidence do I really need? Okay, so we formulated and calibrated. What's next? Step three, compute. This is where you calculate a test statistic. Now, look at this formula here for Z. Don't worry about memorizing it right now. All it's doing is measuring how far your data is from that boring null hypothesis. How weird is your result really? And that brings us to the
1:09
final step. Step four, decide. This is the moment of truth. You compare your p value to your alpha. If your p value is less than alpha, your result is just too weird to be random chance. You get to reject Hnot. Your new theory wins. But if the p value is greater than or equal to alpha, well, it could just be random noise. You fail to reject Hnot. and the boring status quo remains. Now, you've probably heard of a whole zoo of different tests. And it seems really overwhelming. For
1:33
example, in this table, you see that for continuous data with a large sample size, you use a Z test. For a small sample, a t test. And for categorical data like clicks, you'd use a kaiquare test. But here is the secret, the most important thing you need to know. The algorithm never changes. The only thing that changes is the syntax, the specific formula you plug into step three. So, now that we have the framework, let's build some real deep intuition. We'll begin with
1:54
step one using the single most intuitive example there is, the potentially rigged coin. All right, let's start with a bold claim. Forget the formulas and for now, forget the jargon. We're going to build our entire intuition around one simple, powerful question. Is your friend cheating or are they just unlucky? Imagine this scenario. Your friend comes up to you and says, "Hey, this coin is 100% fair, 50/50." So, you decide to test it. You flip it 10 times and the result?
2:22
Well, look at this. Head, head, head, another head. All the way down, you get nine heads and just one tail. Right about now, your internal BS detector is absolutely screaming. But how do you prove it? Well, the methodology is actually super intuitive. Let's walk through this logic. Step one, we're going to start by assuming the coin is actually fair. We'll give your friend the benefit of the doubt. Step two, then we calculate the probability of getting a result this weird purely
2:51
by luck. And that leads to the big question. Is that probability too low? If the answer is yes, we call BS. If the answer is no, well, we have to accept the result. So, let's do the math again. The scenario, your friend claims the coin is fair, but you got nine heads and one tail. If the coin is truly fair, meaning the probability of heads is 0.5, what are the odds? First, what's the chance of getting exactly 10 heads? That's just 0.5 to the 10th power, which is about 0.0097.
3:23
Tiny. Okay. What about the chance of getting exactly 9 heads? Well, that's 10 * 0.5 to the 10th power, which comes out to about 0.0976. So, the total probability of getting 9 heads or more is just the sum of those two. 0.000097 plus 0.00976 gives us our magic number 0.017. That's it. There is only a 1.07% chance that this happened because of pure luck. So now we have our evidence. Our 1.07% is what we compare against
4:01
our BS threshold. And in science that's usually set at 5%. So is 1.07% 07% less than 5%. Yes, it is. The result is impossible enough to reject the fair coin idea. Your friend is cheating. Now, here's the aha moment. What you just did, that entire intuitive process is hypothesis testing. You just used different words for it. This table here is like the Rosetta Stone, mapping your intuition to the formal syntax. That thought, assume the coin is fair, that's called the null
4:33
hypothesis or Hnot. The idea that your friend is rigged, that's the alternative hypothesis or H1. That 1.07% chance we calculated, that has a special name, the P value represented by a lowercase P. And finally, your 5% BS threshold, that's called the significance level, represented by the Greek letter alpha. And this brings us to the one universal rule that governs everything. If P is less than alpha, you reject Hnot. That's the whole game. You can even visualize what rare looks
5:03
like. Imagine a bar chart of all the possible outcomes. The chances of getting 9 or 10 heads would be these tiny little bars way off to the side. That's our rejection region. Of course, in the real world, you're not doing this with a calculator. You're using code. Here's how you'd solve this exact problem in Python. First, the setup. We set K= 9 for our successes or heads, N= 10 for the total trials or flips, and P= 0.5, which is our null hypothesis of a fair
5:30
coin. Then we run the test. We use the function binome test from spyai.stats. We pass in k, n, and p. And notice this last part, alternative equals greater. That's us asking the computer, what are the odds of getting 9 or more heads. And when you print the result, look at that p value 0.0107. The exact same number. And just like that, you've mastered the logic. But here's the next challenge. What about data that you can't count? A coin flip is what we call discrete data. You
6:01
can count the number of heads, 1 2 3 and so on. But you can't really count things like revenue, which could be $1542, or time on a site, which might be 4 minutes and 32 seconds, or the weight of a product, like 12.05 g. This is continuous data. To handle both kinds of data, we need a universal yard stick, a single tool that can measure anything. And that brings us to our next chapter, the Z test. Let's make this concrete with the chip factory problem. Imagine you run a
6:30
factory that makes potato chips. Your target is for every bag to weigh exactly 100 g. Now you know from years of data that the machine has a natural variation, a standard deviation or sigma of 5 g. So you pull a fresh sample of 50 bags off the line and you find that the sample average, what we call Xar, is 102 g. Here's the million-doll question. Is the machine broken and making heavier bags or
6:52
is this 2 g difference just random noise? Guessing is not an option. But you might be thinking, what if the data is weird? What if the weights of individual bags aren't a perfect bell curve? And here is the single most important concept in all of statistics. It's called the central limit theorem. And it makes a beautiful promise. Look at this diagram. On the left, you see the raw data. Your individual bags might be chaotic, skewed, messy, weird. It doesn't matter because if you
7:16
take the averages of samples, like the average of our 50 bags, those averages will always form a perfect predictable bell curve every single time. This is our secret weapon. Because of this, we can use a simple formula to measure how surprising our result is. The intuition is all you need to remember. Think of it as a ratio. Z equals signal divided by noise. The signal is how far off we are. That's our sample average, Xar, minus the target, mu. The noise is how much variation
7:40
is normal. That's the standard deviation, sigma, divided by the square root of our sample size, n. All this formula is really asking is how many standard deviations away is this result. That's it. So, let's run our chip factory problem through the four-step algorithm. Step one, formulate. Our null hypothesis, Hnot, is that the machine is fine. The true mean, mu is 100. The alternative hypothesis, H1, is that the machine is broken and mu is not 100. Step two,
8:04
calibrate. We'll stick with the standard industry threshold. Our alpha is 0.05. Now for the fun part. Step three, compute the zcore. Let's plug in the numbers. The signal is our result 102 minus the target 100, which is 2. Easy. The noise which we also call the standard error is 5 / the<unk> of 50 which comes out to about 0.707. So our zcore is 2 / 0.77 which is approximately 2.83. Wow that
8:29
means we are 2.83 standard deviations away from the target. So is that a lot? Let's visualize it. If you imagine a map of all possible outcomes, a zcore of 2.83 is way out there in the tail. It's deep in the this is really unlikely zone. And that brings us to the final step. Step four, decide. We take our zcore of 2.8 83 and convert it into a p value. That p value is 0.47. Our alpha threshold
8:52
remember was 0.05. So the final question is 0.47 less than 0.05. Yes, it is. Our conclusion, we reject Hnot. This result is too weird to be random chance. The machine is broken. Shut it down. Of course, you're never going to do this by hand. Here's the 1 minute Z test in Python. First, the inputs. Mu is 100, sigma is 5, n is 50, and our observed average xar is 102. Second, we compute the signal and noise. Just like before, signal equals xar minus mu. Noise equals sigma
9:22
divided by numpi roo<unk> of n. Z equals signal divided by noise. Finally, we get the p value. We use a function from scypi called norm.cdf. The two times at the front is because it's a two-tailed test. We'd be surprised if it was too high or too low. And when we print the result, look at that. Z= 2.83 83 and P= 0.47. The exact same numbers. The code confirms our decision. But there is a huge flaw in what we just did. We cheated. Here's the impossible assumption. In our
9:50
factory, we knew that sigma was 5 g. We had years of historical data to tell us that. But in a new AB test, how could you possibly know the standard deviation of the entire population? It doesn't exist yet. You haven't rolled out the feature. You can't know it. You have to estimate it from your small sample. And adding an estimate adds uncertainty. To handle that extra uncertainty, we need a slightly different, more powerful tool. And that brings us to the next chapter
10:11
and the true workhorse of all AB testing, the TA test. All right, let's set the scene with a classic scenario. We'll call it the new button. Our goal is to increase the time users spend on our site. We're running a really small test, just 10 users in each group. For group A, who saw the old button, their average time was 45 seconds with a sample standard deviation of 15. For group B,
10:31
who saw the new button, their average time was 55 seconds, also with a standard deviation of 15. So, we got a 10-second increase. That sounds great. But with only 10 users, is it real? To figure that out, we need our new tool, the t statistic. And here's the good news. It's still just signal divided by noise. Look at this formula. The signal on top is just the difference in our means, xarb
10:52
minus xar a. And the noise on the bottom, well, it looks a bit scary, but it's just combining the uncertainty from both samples. And notice the key detail. We use S, the sample estimate, not sigma, the absolute truth. That's the whole point. Okay, let's run the four-step playbook. Step one, formulate. Our null hypothesis, Hnot, is that there's no real difference. The new button does nothing. Our alternative hypothesis, H1, is that the new button does work, that the true means are
11:17
not equal. Step two, calibrate. We'll stick with our industry standard threshold. Alpha is 0.05. Now, for the math. Step three, compute the t-core. First, the signal. That's easy. 55 - 45 is 10. Next, the noise. We take the<unk> of 15^ 2 / 10 + 15^ 2 / 10. That's the<unk> of 22.5 + 22.5, which is the<unk> of 45, which comes out to about 6.71. So, our final treore is signal / noise. 10 / 6.71,
11:47
which is about 1.49. Okay, this is the moment of truth. Step four, decide. Our t-core is 1.49. When we convert that into a probability, we get a p value of approximately 0.15. Our alpha threshold, remember, was 0.05. So the big question is, is 0.15 less than 0.05? No, it's not. What's our conclusion? We fail to reject Hnot. That 10-second improvement, it's likely just random noise. 10 seconds wasn't enough. So why did we fail? Let's look at the numbers again.
12:17
Our t-core was the signal of 10 divided by the noise of 6.71. The noise was almost as big as the signal. And that's because small sample sizes like our n equals 10 generate massive noise. So how do we fix it? How do we win this test? Well, you've really only got two levers to pull. Option one, get a huge signal. Make an improvement so gigantic that it just completely dwarfs the noise. Or the
12:38
more common path, option two, get more data. Increasing your sample size is the single most powerful way to shrink that noise term. Of course, in the real world, you just use code. Here's what the production code looks like. We import stats from scypar. We've got our two lists of data, group A and group B with the actual time onsite values for each user. Then we call one single beautiful function stats.test_indd passing in our two groups. And look at the output it gives us. T=
13:03
2.29 and P= 0.0342. Now wait for this data. The P value is less than 0.05. So with this particular data set, it would be a success. We'd have a winner. Okay. So with the t test in our back pocket, we are now officially dangerous. We can handle continuous variables, things like time, weight, money. But what about the most common question in business? What about clicks? Did they click the button or not? That's a yes or no question. You can't average that. For that kind
13:28
of data, we need one more tool. In the next chapter, we'll cover the kaiquare test. Let's imagine a classic scenario. We'll call it the red button test. You're running a website and you're testing a blue buy now button versus a red buy now button. You send 100 visitors to each. In group A, with the blue button, you get 10 sales. That's a 10% conversion rate. Not bad. But in group B, with
13:47
the red button, you get 20 sales, a 20% conversion rate. It doubled. The answer seems obvious, right? Ship the red button. But hold on. With samples this small, how do we know this isn't just a lucky fluke? Is it real? To answer that, the Kaiquare test gives us a brilliant new mental model. Forget signal versus noise for a moment. Instead, think about observed versus expected. Observed,
14:07
which we'll call O, is simple. It's what actually happened, the raw data you collected. Expected or E, is what the null hypothesis predicts. It's what would have happened if your change made no difference at all. The logic is simple. If there's a big gap between what you observed and what you expected, that's a real effect. If it's a small gap, it's probably just noise. So, how do we measure that gap? We use the K square distance formula. Now, this looks a little scary,
14:28
but I promise the idea is super simple. K^2 equals the sum of O minus E^2 / E. Let's break that down. First, O minus E. We just find the error or the difference between what happened and what we expected. Second, we square it. This is clever. It makes all the errors positive and it really punishes big mistakes. Third, we divide by e. This normalizes everything. A difference of five is a huge deal if you only expected 10, but it's a rounding error if you expected 10,000.
14:51
Context matters. And finally, we sum it all up. We just add up the normalized squared error from every single cell in our experiment. Okay, let's apply this to our red button test. We have four steps. Step one, formulate the hypothesis. Hnot is that color doesn't matter. H1 is that color does matter. Easy. Step two, build the observed table. This is just our raw data. For the blue button group, 10 people bought and 90 did not. For the red button group, 20 people
15:14
bought and 80 did not. That's it. That's what we observed. Now for the fun part. Step three, build the expected table. What would this table look like if the null hypothesis were true? If button color really didn't matter. Well, overall, we had 30 sales from 200 total visitors. That's an overall average conversion rate of 15%. So, if the null hypothesis is true, we would expect 15% of people in the blue group to buy. That's 15 sales. And we'd expect 15% of people in the red
15:39
group to buy. Another 15 sales. See under the null hypothesis we expect the groups to be the same. And now step four compute k square. We just plug the numbers from our observed and expected tables into the formula. For the blue group who bought we observed 10 but expected 15. So that's 10 - 15^ squar / 15 which is 1.67. For the red group who bought, we observed 20 and expected 15. So that's
16:06
20 - 15^ 2 / 15 which is also 1.67. Then we add in the errors from the non- buyers and our total kaiquare value comes out to about 3.92. When we convert that score to a p value, we get 0.0477. So the moment of truth is 0.0477 less than our alpha of 0.05. Yes, just barely. Our conclusion, we reject the null hypothesis. The red button really is better. Of course, in the real world, you are never going to do this by hand. You'll use code. You import kai2 contingency from
16:39
cypit.stats. You create a simple list of lists for your observed table. Just 10 and 90 for the blue group, 20 and 80 for the red group. You pass that to the function. And look at that output. Kai2 is 3.92 and the p value is 0.0477. A perfect match. We won. The data proved it. The red button is better, right? But what if one single customer had changed their mind? Let's do a reality check.
17:05
In our original test, red got 20 sales, giving us a p value of 0.0477. That was significant. Now, imagine a parallel universe where the red button got just one less sale, 19 sales instead of 20. When we rerun the numbers, our p value jumps to 0.0763, which is not significant. One single person flipped the entire decision of the AB test. This reveals the most important truth about this kind of data. Categorical data is low resolution. It's just a yes or a no. There's no
17:36
nuance. And because of that, you need huge sample sizes to be confident in your results. All right, you've made it. You battled through Z tests, t tests, and kaiquare. You've seen the universal algorithm in action. But now we move from theory to production. This is the most important part because here's the first thing you need to understand. You aren't a human calculator. That's not your job. Your real job is decision-making. It's about knowing which tool to use,
17:56
when to use it, and how to interpret the results without fooling yourself. So, let's pull back the curtain and look at the God view. This is the decision matrix. If you walk away with one thing, let it be this. You just look at your data, ask the right question, and pick your tool. It's that simple. For example, in this table, let's say your data is categorical, things like clicks or a simple yes or no. The question you're asking is, is the conversion rate different? Your tool,
18:14
the kaiquare test. Easy. But what if your data is continuous like time or revenue, and you're asking is the average different, but you don't know the population standard deviation, which is like 99% of the time. Your tool is the two sample t test. And for those rare cases in manufacturing or science where you have continuous data and you do know the historical population standard deviation, you ask, "Is the average offtarget?" And you use the one sample Z test. That's it. That's
18:34
the playbook. But knowing the playbook isn't enough. The pros know the traps. Let's talk about the first production pitfall. P hacking. Here's the temptation. You launch your AB test. You're excited. You check the dashboard every hour. Hour one, the P value is 0.30. Boring. Hour two, it's 0.15. Ooh, getting closer. Hour, it's 0.049. Yes. Stop the test. Ship it. You just cheated. you
18:55
did because P values are noisy. They bounce around like crazy, especially at the beginning of a test. If you keep peeking, you're basically waiting for a random fluctuation to cross the finish line. It invalidates all the math. So, what's the professional solution? It's a simple four-step process. One, you pre-commit. You say, "I need 10,000 users for this test, not 9,999. Two, you run the test." And this is the hard part. You keep your hands off the dashboard. Three,
19:15
no peing. Seriously, it's the cardinal sin of AB testing. And four, you decide once. You check the p value only at the very end when the sample size you committed to has been reached. Okay, that's the first trap. Now for the second, more subtle one. Production pitfall number two. I call it the Google problem. Here's the scenario. You change one tiny line of code at Google scale. Your sample size is 50 million users. The result, you get a 0.001 second improvement in search speed.
19:45
And because the sample size is astronomical, your p value is less than 0.00001. 00000000001. So do you launch? This is where you have to understand the two types of significance. First statistical significance. It asks is the effect real? And in this case, yes, we are 99.999% sure it's not random noise. The math is screaming at you. But then there's practical significance. It asks a better question. Does it matter? And for our Google problem, the answer
20:21
is no. A 0.001 second improvement is completely invisible to humans. It provides no real value. So remember this, a tiny p value does not mean a big effect. It just means that we are very very sure that an effect exists. It could be a tiny useless microscopic effect, but we are certain it's there. This leads to the production rule. You always check two things before making a decision. First, the p value. This tells you, does the effect actually exist? It's about certainty. Second,
20:56
the effect size. This tells you, is the effect big enough to care about? It's about magnitude. A good decision requires both. And with that, your 20-minute journey is complete.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat