Skip to main content
Loading Explore guides…
Hypothesis Testing · Acedra
Home
Explore
Statistics
Hypothesis Testing
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
215 segments
Copy
Hide video
0:00
now let's talk about hypothesis testing
0:06
hypothesis testing is harder than confidence intervals at the most basic level confidence intervals you could think of as being fairly easy actually because you just do the calculation say there's the answer although interpreting them and understanding what they represent is a little trickier for people hypothesis testing is definitely harder
0:29
and let's start it uh by thinking about an example let's go back to my terrible free throw shooting okay so you know that I've told you that I think I'm about a 40 free throw shooter terrible right basketball players of which I was one back in sixth grade should be like at least 60 percent free throw shooter that would be ideal but better be 80 or something
0:54
um but that was not very good at shooting I I'm still bad at shooting I can't make it into my waste basket in my office either very well and uh so I think I'm a forty percent shooter basically not I haven't really done tons of experiments I've done some experiments with it it seems about right but you think you know Dr Kenny I think you're even worse
1:16
I think you're a 20 free throw shooter okay you have a hypothesis called the null hypothesis
1:24
H 0 that you think my probability of making it on any particular attempt because you've you've seen me and you've seen how bad I am you think it's 20 or maybe even less
1:40
p is the probability that I make it on any one free throw attempt
1:46
I have an alternative hypothesis I think I'm a 40 free throw shooter but you you say I'm going to make it easy on you Dr Kenny if you just prove that you are better than a 20 free throw shooter then I won't press you any further the alternative hypothesis is that P equals is bigger than 0.20. our book uses H Sub Zero or H naught
2:13
zero sometimes called not not not don't say h naught say h naught for the null hypothesis
2:25
null hypothesis and H1 for the alternative hypothesis which is what I think is true you think the null hypothesis is true I think the alternative hypothesis is true
2:41
other books do use different notation of the alternative hypothesis H sub a a for alternative
2:48
okay it's not a big deal that's what our book uses what is p p is the parameter that we are testing here p is my true probability of success success being I make the free throw
3:10
now you might wonder does that probability of success stay constant well in an idealized situation it seems reasonable to assume it's constant what's idealized like well if I'm in a gym and I'm not tired and there's nobody watching me right though if I'm outside and it's windy and people are watching me a lot of pressure on me okay then maybe it's different
3:35
game situation you might wonder if it's different than practice situation whatever
3:42
that's my true probability of success it's a population parameter it's unknown
3:52
so again I'm pretending you think the null hypothesis is true I think the alternative is true in fact I think a little bit more precisely the alternative is really that P is 0.4 but again you're saying Dr Kenny if you can just prove p is bigger than 0.2 I'll give it to you okay
4:11
so what do you do based on this setup of the hypothesis test you need to actually do a test now this is the setup of the hypothesis test
4:21
how do you actually do a test you have to do an experiment
4:29
do a random sample shoot some free throws I stand there do some free throws how many should I do I guess that would be the sample size
4:40
I guess I could do any sample size though it should seem a little reasonable hopefully reasonable that the bigger the sample size maybe the the more uh what's the best word uh
4:55
accurate the test is the bigger the sample size although again there's questions about would I get tired right if I'm going to do a thousand free throws I'll get too tired
5:05
maybe I do 10 free throws a day for 100 days and I think of that as one big sample of a size 1000 or something but to make it easier to think about let's make the sample size a little smaller
5:20
let's make it size n equals 20 say
5:28
um
5:30
so I do it I do the random sample I shoot 20 free throws I count how many I make let X be the number that I make let x equal the number of successes
5:41
so I'm just saying success is here but I'm really thinking about free throws
5:48
what kind of random variable would X be
5:56
this should be pretty obvious somebody say it what kind of random variable is this binomial yeah
6:07
with n equal to 20 and P unknown really who's right you or me we don't know for sure
6:18
Which hypothesis is right we don't know for sure
6:25
now if P truly is 0.2 or smaller say p is 0.2 you would expect in 20 attempts that I would get four or fewer successes four is twenty percent of 20.
6:44
so if I get four or fewer successes that's evidence for the null hypothesis being right if I get five or more successes that's some evidence for the alternative hypothesis being right
6:58
certainly if I make all 20 free throws I'm right right there's pretty much no chance you're right
7:05
p is bigger than 0.2 it's like a miracle occurred okay that would be very strong evidence against the null hypothesis even if I make 10 out of 20 that's pretty strong evidence against the null you might even guess even if I make eight that seems like pretty strong evidence against the null but what if I only make five out of 20 that's 25 am I right well maybe I was just lucky
7:35
that particular run of 20 attempts I happen to make five I just lucky
7:43
if I make six yeah I can still see it happening even if p is 0.2
7:48
make thirty percent once you start to get to seven or eight above then you're like okay that seems like strong enough evidence against the null to reject it but we need to quantify that
8:03
there's a couple different ways to quantify it to decide whether to reject the null hypothesis in favor of the alternative hypothesis the classical way is what I'll show you first
8:16
is to decide on a here's the word again level of significance ahead of time
8:23
and Alpha
8:26
and let that Define what's called a critical region
8:30
or rejection region value of x maybe it's if I make seven or more I reject the null
8:40
alternatively I could also compute something called the wait for it the most controversial subject in statistics p-value of the test
8:50
heard of p-values before nobody raise your hand no okay a few people penis it's the most controversial subject in statistics and applied statistics is use and abuse of P values in studies
9:09
what's a p-value
9:13
I'll say it now I will say it again and people always almost always get this wrong when they're done with Statistics classes even scientists and engineers get this wrong the p-value of the test
9:28
is the probability P stands for probability of observing what you actually observe
9:37
when you assume the null is true
9:41
I actually have to think about it as a probability ahead of time but then you compute it after the fact so you get a certain number of successes
9:50
I didn't quite say that as precisely as I should have it's the probability of observing
9:56
What You observe or more extreme under the assumption that the null is true
10:07
I'll be more precise when we when we actually computed here okay
10:12
if that probability is small really small what that means is You observe something rare if the null is true therefore you reject the null
10:26
here's the most common misinterpretation of the p-value people think it's the probability that knows people think the p-value is the probability that the null is true that is wrong
10:41
nobody but nobody in the world can determine the probability that the null hypothesis is true it's impossible
10:50
I will re-emphasize these things and I probably will write them down as well just not right now
10:56
let's do it the classical way first
11:00
decide on
11:04
classical approach by classical I mean uh like 40 years ago or earlier
11:16
right 40 years ago was a long time in statistics
11:20
lots of Statistics has been done in the last 40 years compared to before that classical approach
11:29
right when we're doing statistics it's almost all math from the last hundred years
11:34
whereas when you do things like calculus that's like two or three hundred years old doing geometry it's a couple thousand years old classical approach is uh choose a level of significance
11:55
Alpha
11:58
which represents a probability it represents the probability of making something called a type 1 error when you do the test
12:08
you are presetting your probability of making a type 1 error but what is a type 1 error
12:17
you could think of it as a conditional probability the probability of rejecting the null
12:25
if you assume the null is true
12:33
that's an error you don't want to reject an elf until it's true
12:37
you don't want to say the null is false when it's really true right that would be a mistake
12:45
it's not a math mistake it's a decision mistake it's like in a courtroom it's like convicting an innocent person the null hypothesis is that the person's innocent and in our court systems right that's what we assume innocent until proven guilty the alternative hypothesis is they're guilty
13:07
rejecting the null is like saying they're guilty the null is true means there really were innocent
13:14
so you made a mistaken decision of sending an innocent person to jail that's a type 1 error
13:23
as a society we we typically think of that as being worse than a type 2 error which is letting a guilty person go free acquitting them
13:34
right which is worse sending an innocent person to jail letting a guilty person go free well people might have different opinions about which one's worse but since we give the defendant the benefit of the doubt and assume they're innocent the burden of proof is on the prosecution
13:51
we want it hard to make a type 1 error
13:57
that means we're intuitively thinking about type 1 error as being worse
14:03
certainly it's not good to let a guilty person go free stuff you know terrible crime like murder or something but you can't avoid it sometimes so what do you choose for the level of significance
14:19
a common thing to use is point zero five five percent do you have to no
14:27
people are kind of stuck on five percent that's part of the controversy with P values by the way people are stuck on five percent that's the magic number
14:38
and I will use five percent here for the sake of illustration but you don't have to use five percent you could use one percent two percent three percent seven percent
14:47
9.45 if you wanted to
14:53
it's up to you how do you decide
14:58
the smaller you make alpha the harder you're making it to make a type one error so if a type 1 error is really bad like people will die if you make a type 1 error
15:07
then you want to make alpha really really small I mean literally if people die if you're making a type 1 error you don't have to be really really small
15:19
you can't pick zero though right that's not really a plausible thing to take you got to pick something positive maybe it's really really small like point zero zero zero zero one or something absolutely
15:34
but it's always a trade-off if you make it harder harder to make a type one error harder to uh connect an innocent person you're making it easier to let a guilty person go free you're making it a better chance of making a type 2 error
15:55
no way to avoid it
15:58
uh or is there
16:02
it depends on what you mean you can make things better in statistics on both counts by making your sample size bigger make end bigger
16:16
so you choose the level of significance for the sake of illustration
16:21
let's go ahead and use the five percent
16:26
second thing to do with the classical approach is uh determine the critical region determine
16:33
the critical region
16:39
also known as aka the rejection region
16:47
what's that that's the set of values of X observed values of X that will lead to rejection
16:59
you got to use the level of significance to figure it out
17:03
and how do you figure it out
17:07
well we got to calculate that probability
17:15
and figure out what boundary value of the region will lead to this being approximately 0.05
17:26
but how do you do that I'm calculating this probability assuming the null is true the no being true includes infinitely many possible values of P
17:35
less than or equal to 0.2 how am I going to do that choose the boundary value
17:45
choose
17:48
I'll call it a boundary value
17:52
assume for the sake of argument that P is 0.2
17:58
that's called a simple hypothesis so if I go ahead and do that I'm really rewriting the excuse me the null hypothesis to be inequality
18:10
now in reality it's not really inequality it's an inequality
18:17
this is called a simple hypothesis make it an equal sign why is that okay
18:25
because if I compute the p-value for that simple hypothesis the probability of rejecting the null if p is exactly 0.2
18:37
that'll be the highest possible probability for a type 1 error
18:46
because if P were even smaller like 0.1 there'd be even less chance
18:54
of being in the critical region to reject the null if P really is 0.1 say compared to point two there there be a if p is 0.1 instead of 0.2 there's a small smaller probability that X is say bigger than or equal to seven
19:11
then if p is 0.2 I got a better chance of making seven or more free throws if P really is 0.2 then I do when p is 0.1 so I'm really Computing this in the the most extreme case where it's as large as possible
19:26
that's what I'm thinking about so if I reject the null in that case I'll also reject it in if P really is 0.1 say if that was too fast in an explanation um I can explain it again later
19:44
so what do we have to do here um I'm wondering in the certain situation
19:53
what value of x I'll make it a question mark we'll make this probability equal to approximately
20:04
0.05 I need to find a question mark find this
20:13
there is my assumption that the null is true p is 0.2 based on the simple hypothesis
20:20
what number could I put there that will make this probability approximately 0.05 that's what I'm after is it seven is it eight we don't know yet let's see if we can figure it out uh this would be the same
20:36
as the probability that X is strict or excuse me as one minus the probability that X is strictly less than question mark given that p is 0.2 and so we could use the CDF function on the calculator
20:55
and probably we just need to guess
21:01
I mean if we had a table available to us which we do
21:05
we could avoid guessing but with calculator we'd have to guess let's just guess seven and see what happens so yeah let me go one minus distribution menu
21:20
binomial CDF n is 20. p is 0.2 I think p is 0.4 but the null is that P is 0.2 if I put a 7 there
21:34
oh and careful okay I gotta be careful because that's a strict list then that's the same as 1 minus the probability that X is less than or equal to question mark minus one
21:48
right because X is discrete and can I take on all the whole number of values
21:56
so if I put a 7 here I'm really figuring out I'm really assuming this is seven question mark minus one so I'm really figuring out when the question mark is eight let's just see what it is
22:08
0.03 pretty close to 0.05 that we're pretty close to the right answer
22:16
if I do let's just test the um one number smaller
22:24
six
22:28
a little bit bigger than .05 so we don't get exactly 0.05 for either of these that's because X is discrete they're going to play it safe I better use the seven so it seems like question mark minus one should be seven so that question mark is eight and what this means is the critical region
22:51
also called the rejection region consists of the numbers 8 9 10 Etc up through twenty if I make eight or more free throws we're going to reject the null in favor of the alternative we're going to say that P is bigger than 0.2 in our uh probability of making a type 1 error technically is a little smaller than .05 it was 0.03 something
23:17
I was just after an approximation here
23:24
then you actually do the test I go ahead and shoot my free throws maybe I make less than eight then we'd fail to reject the null maybe I make eight or more then we would reject them you make your decision I'll count that as step three do the experiment
23:45
and make a decision
23:53
reject the null or not
23:57
when you reject the null reject h0 is that the same as accepting the alternative except H1
24:09
although I will say that most people when they do the test just say we reject the null
24:18
it's kind of like not always sometimes people say accept the alternator but it's kind of like you're the prosecuting attorney when you do these tests
24:29
you think that you're me you think the null is wrong and you're trying to prove it wrong
24:37
okay uh gotta be careful I guess Virginia you're wanting to reject the null
24:47
question yeah it's like you're me the alternative is that I'm worse for the nulls that I'm worse the alternative is that I'm better
24:58
usually in a courtroom it's the opposite way yeah the null that they're innocent is the better thing for the defendant at least and the alternative that they're guilty is the worst thing so usually in a courtroom
25:14
you're trying to the the prosecuting attorney is trying to reject the null maybe they really think that defendant is guilty they want to keep their job they want to reject the null they want to the court to say they're guilty
25:34
and I'm getting this next up here
25:42
okay but if
25:46
okay
25:48
if they fail to reject them though
25:53
it doesn't mean they accept them right it just means they fail to have enough evidence against the mill to convict the person that's what I meant to say
26:06
so you either say reject the null which basically does mean except the alternative
26:11
or fail to reject the null which people do not typically say is acceptable
26:19
you just don't have enough evidence against it it's like you're the prosecuting attorney you don't think that person really is innocent you just didn't convicted
26:28
you didn't have strong enough evidence to convince the jury that's what I meant to say
26:35
okay that's the classical approach what's the um modern approach and by modern I mean the last 40 years
26:56
uh the modern approach is to uh
27:00
compute the p-value right away
27:08
that's step one once you set up your hypothesis test
27:17
once you've got the p-value second step is make a decision is the p-value small enough for you to reject them all or not done
27:28
in both of these situations you do need to think about the practical consequences of your decision maybe it leads to some business decision like we're gonna invest in this new marketing campaign or not
27:46
so you need to actually apply it in real life and not just do the theory decisions need to be made based on either rejecting the null or failing to reject the null but what is the p-value here
28:01
the p-value in general writing it down what I said about 20 minutes ago
28:09
but I'm going to be a little bit more specific it's the probability
28:15
of observing
28:19
a test statistic value
28:27
as extreme or more extreme
28:36
than what you actually observed
28:44
under the assumption that the null hypothesis is true
28:54
onto the assumption that h0 is true it's a theoretical calculation
29:00
that leads to practical decisions
29:07
what is a test statistic and what is as extreme or more extreme mean
29:12
the test statistic varies depending on the situation
29:16
for our example here effectively the test statistic is the observed value of x
29:22
though it could be modified to be an observed value of a sample proportion x divided by n
29:34
as an estimator for p
29:37
if I make eight free throws out of 20 then I made forty percent point four it could be that as well
29:46
that's what we do actually at the beginning of chapter nine next Tuesday
29:51
it could be X or it could be when we're testing means as we will in a few minutes a value of x bar
30:01
or we could convert the value of x bar to a z statistic or a t statistic
30:11
so in our case
30:18
we call it a p-value let me just pretend I do the experiment and I make six free throws
30:31
what's the p-value for that test if I assume I make six out of 20.
30:36
if the probability that X is greater than or equal to 6 as extreme or more extreme with respect to the direction of the null of the alternative hypothesis being a greater than
30:50
to the right I need to compute an area under something to the right of something
30:56
under a distribution to the right of six in this case if I make six I do this calculation assuming the null is true assuming p is 0.2
31:07
and that would be the same as this
31:12
this should be bigger than .03
31:20
I forgot was this one let's do it again 1 minus
31:25
binomial CDF
31:31
okay I need to use five here so I'm going to get bigger than .08 as well
31:39
it's about 0.196 if I make six free throws and 20 attempts the p-value is 0.196 for that test
31:54
this represents the probability that I'm going to make six or more free throws in 20 attempts if my true probability of success is 0.2 now again calling this a probability is a little bit funny because it's after the fact
32:11
whereas I said probably should be really thought about before the fact but I mean you could go back and think of this before the fact maybe I'm about to do the experiment again and I'm wondering really if my probability of success is 0.2 and I do 20 attempts what is the probability that I make six or a month now that's a small number less than 0.5 less than 0.2 even
32:33
but it's not really really small it's not small enough for most people to reject the null
32:40
so we'd say fail to reject the null compute the p-value make a decision it's the second step
32:49
and if that were my p-value in a
32:55
standard kind of situation people would say that's not small enough to reject so we fail to reject the null
33:04
again it's not that you're saying we accept the null not that I'm saying acceptable I really think I'm better than 20 shooter I just don't have enough evidence to reject it because this is not while it's small it's not really small what's really small less than .05 is the standard thing people use but it's a matter of opinion sometimes
33:35
make a decision maybe make based on that decision
33:40
to reject or not another decision in business settings to actually like invest in something or not
33:49
interpret what it means practically I I don't have enough evidence to reject the mill
33:57
I observed something somewhat rare but not really rare if the null is true and you give them all the benefit of the doubt just like we assume the person's innocent and we don't reject it
34:09
if you said reject the null on a test problem when you got this p-value would I mark it as wrong
34:16
um I'd say that's not a good idea and yeah I probably would take points off
34:22
you want to be for sure I think less than 0.1 some people say less than 0.05
34:29
if it's not that you can't talk about Alpha in this context you can still talk about the level of significance and if you say Alpha is .05 and that's the alpha you want to use and this is bigger than .05 then you have you definitely should not reject the null you would only reject it if it's less than .05
34:50
you say this is not statistically significant at the five percent level if you get something less than 0.05 then you would say it is statistically significant to five percent low
35:05
a little bit more about terminology the classical approach is often called hypothesis testing this approach which really is still hypothesis testing is sometimes called significance testing
35:17
it's when you use the phrase is this statistically significant at the five percent or one percent level or whatever you're using for Alpha
35:33
there's a lot going on here there's even more that I haven't talked about yet
35:40
you got a parameter you're testing you've got to set up your hypotheses technically speaking the null and alternative neither one in reality is simple but to do the test you got to assume the null is simple
35:53
you can use one of these two approaches both of them can involve thinking about Alpha the probability of the type 1 error and wanting that to be small
36:05
you can think either in terms of critical regions or in terms of P values and by the way with P values again people will often misinterpret this and think it's the probability that the nulls true
36:17
20 chance the probability the null is true no no don't ever think that nobody but nobody can figure out the probability the null is true it's impossible
36:30
I guess we're out of time I forgot we end at 1 30 I thought the class went to 140. I'd like I got 10 more minutes oh I can't assign as many homework problems as I was thinking of
36:43
there I do want you to read sections 8485 and maybe I'll assign one problem or two based on the reading that I haven't gotten to for testing means one of which involves using a t statistic I think I will assign one or two problems based on the reading but I haven't gotten to it in class
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat