Skip to main content
Loading Explore guides…
Simple Linear Regression · Acedra
Home
Explore
Mathematics
Simple Linear Regression
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
70 segments
Copy
Hide video
0:00
(gentle acoustic guitar music)
0:17
- [Brandon] Hello, thanks for watching, and welcome to the next video in my series on basic statistics. Now as usual, a few things before we get started. Number one, if you're watching this video because you are struggling in a class right now, I want you to stay positive and keep your head up. If you're watching this, it means you've accomplished quite a bit already. You're very smart and talented, but you may have just hit a temporary rough patch.
0:42
Now I know with the right amount of hard work, practice, and patience, you can work through it. I have faith in you, many other people around you have faith in you, so so should you. Number two, please feel free to follow me here on YouTube, on Twitter, on Google Plus, or on LinkedIn. That way when I upload a new video, you know about it. And it's always nice to connect with my viewers online. I feel that life is much too short and the world is much too large
1:12
for us to miss the chance to connect when we can. Number three, if you like the video, please give it a thumbs up. Share it with classmates or colleagues, or put it on a playlist. That does encourage me to keep making them for you. On the flip side, if you think there's something I can do better, please leave a constructive comment below the video, and I will take those ideas into account when I make new ones. And finally, just keep in mind that these videos are meant
1:39
for individuals who are relatively new to stats. So I'm just going over basic concepts. And I will be doing so in a slow, deliberate manner. Not only do I want you to know what is going on, but also why, and how to apply it. So all that being said, let's go ahead and get started.
2:02
Okay, so this is the first video in what will be, or is, depending on when you're watching this, a multi-part video series about simple linear regression. In the next few minutes, we will cover the basics of simple linear regression starting at square one. And for the record, from now on if I say just regression, I am referring to simple linear regression as opposed to multiple regression or models that are not linear, which we will hopefully get to those at a later date.
2:33
Now regression allows us to model mathematically the relationship between two or more variables, using very simple algebra, to be specific. For now, we'll be working with just two variables; An independent variable, and a dependent variable. The truth is, when we talk about how quote "good" a regression model is, we are actually comparing it to another specific model.
3:02
Oftentimes, students don't realize this. So in this video, we're gonna talk about that idea. I will also begin introducing basic terminology and concepts that will carry you through your work using regression. There are no formulas or calculations in this video. We're just introducing the underlying meaning behind good regression models.
3:28
So if you are new to regression, or are still trying to figure out exactly what it even is, this video is for you. So sit back, relax, and let's go ahead and get to work.
3:44
So as always, I like starting out my videos with a problem, and a relatively real world problem at that. So we'll call this one tips for service. So let's assume that you are a small restaurant owner, or a very business-minded server or waiter in a nice restaurant. Here in the US, tips are a very important part of a waiter's pay. Most of the time, the dollar amount of the tip is related to the dollar amount of the total bill. So if the bill is $5, that would have a smaller tip
4:20
than a bill that is $50.
4:24
Now as the waiter or the owner, you would like to develop a model that will allow you to make a prediction about what amount of tip to expect for any given bill amount.
4:37
So therefore, one evening, you collect data for six meals. So a random sample of six meals.
4:49
But unfortunately, when you begin to look at your data, you kind of forgot something. You realize you collected data for the tip amount and not the meal amount that goes with it. So unfortunately right now, this is the best data you have. So you have a random sample of six meals and the tip amount for each one of those meals. So $5, $17, $11 and so on.
5:13
Now here's the question. How might you predict the tip amount for future meals
5:21
using only this data?
5:24
There's only one variable here, the tip amount. The meal number's just a descriptor. So we have one variable, the tip amount. But I still want to challenge you to come up with a model that will allow you to predict within some reason what the next tip is going to be. How can you do that? Think about it.
5:48
So the first thing we're gonna do is we're going to visualize our data. As you know if you watch my other videos, I am a huge advocate of visualizing our problems, making charts, graphs, diagrams, whatever we have to do to make them visual. So the first thing we'll do is we'll make a graph of our tips. Now on the x-axis on the bottom, we have our meal number. Now that's not a variable, that's just a descriptor of what meal we're graphing. Now on the y-axis, or the vertical axis,
6:17
that's where we will graph our tip amount. Let's go ahead and see what this looks like. So for meal one, with a tip of $5, so we'll go ahead and graph that at around $5. For meal two, with a tip of $17, so that goes way up there. For meal three, with a tip of $11, so that goes there. Meal four, with a tip of $8, that goes there. Meal five, that was a $14 tip. And meal number six, that was a $5 tip. So here are our data points. Remember, we're only dealing with one variable,
6:51
that's the tip amount, and the meals along the bottom just describe where we're graphing each point. And the order does not matter. We could have graphed these in any order. This just happens to be the one we ended up with. Now, what's really the most you can figure out about this data?
7:10
How would you predict what the tip for meal number seven would be? Is it going to be like meal number six, it's $5? Is it gonna be like meal number two, to $17? How would you come up with the best guess or estimate for the next meal using only one variable? Well, you would use its mean.
7:34
So the mean for all six tips is $10.
7:39
So guess what? That's the best we can do. With only one variable, the best estimate for the best prediction, for any given meal tip is $10.
7:52
So go ahead and put a line at $10. So that for this model, that is our best fit line, that's all we have. One variable, tip amount, the mean is the best predictor of any given tip amount.
8:07
Now obviously if you look at this chart, our tips do not fall on the $10 line, they're scattered around it. But still, it's the mean. So that's your best estimate for the next tip for any given tip would be.
8:27
So here's our graph again with our tips, our mean, and our tip amount. I just want to stress that the tip amount is y bar, so that's the mean of y, and that's for two reasons. One, the dependent variable, which it will be as we progress forward is always the y of the x and y axes, and of course we're graphing it on the y-axis, so it should be y bar. So here it is, the basic concept I really want you to remember in your head as you go forward. But obviously, simple linear regression
8:59
is about two variables. But, we're starting off here, 'cause this is where it all begins. With only one variable and no other information, the best prediction for the next measurement is the mean of the sample itself.
9:18
So the variability in the tip amount, 'cause they're not on the line, they're above and below, the variability in the tip amounts can only be explained by the tips themselves because that's all we have. So the way they're above and below the line, that's just the natural variation in the tips. But the basic point is this; With only one variable, the best way, the only way we can make a prediction about what the next tip amount in this case is the mean.
9:50
So our best prediction for the tip of meal of number seven is $10.
10:01
So let's talk about the goodness of fit for this line and our tips. Now obviously we know that the data points, the actual observed values do not fall on that line, they do not fall on the $10 line, some are above and some are below it. So that tells us how good this line fits these observed data points. Now one way we can do that is to measure the distance they are from that best fit line. Now we did this to some degree when we were talking about standard deviation.
10:37
Remember, we're talking about the distance each data point is from the mean. But guess what we're doing here? The distance that each data point is from the mean, because the mean is our line of $10 here. So, for meal number one, our tip was $5.
10:55
so that's $5 below our mean of $10, so that's negative five. Meal number two, got a tip of $17, that was $7 above our mean. Meal three was $11, $1 above our mean. Meal four was $8, that's two below our mean. Meal five was $14, that's four above our mean. And meal six is $5, that's five below our mean. So these are the distances, in this case, dollar amounts by which each observed value is different from or is, deviates from
11:32
the mean of $10.
11:36
Now we have a name for these, they're called residuals. So the distance between the best fit line, which in this case, 'cause it's one variable is $10, the distance from the best fit line to the observed values are called residuals. Now they're also called the error. So the distance is also called the error because that's how far off the observed value is from the best fit line. Now you notice a few more things here. If you add up the residuals on the top,
12:12
just above the line, seven plus one plus four, that's 12. Add up the residuals below the line, five, two and five, so that's minus 12.
12:23
So the residuals always add up to zero.
12:28
Now that's another important concept to keep in mind as we go forward.
12:35
But if you remember in standard deviation, one of the steps was that we took the deviations from the mean and we squared them. Well guess what? We're gonna do the exact same thing here. So the residual for meal one one was $5, so it's $5 below, so we square that and it squares to 25. Meal number two, it was $7 above, we square seven, that's 49. So on and so forth. So the right-hand column of our table, we have our squared residuals. Now the question is why do we square them?
13:10
Well we square them for the same reasons we square the deviations when calculating the standard deviation. Number one, it makes them all positive. So if we square a negative number, it obviously makes it positive. And number two, it emphasizes the larger deviations. So a deviation of two will square to four. But a deviation of five will square to 25. So the squaring really exaggerates the points that are further away. Now what we can do is we can take these residuals,
13:45
these squared residuals in the right-hand column and we can add them up. And they're called the sum of squared residuals, or the sum of squared errors, or the SSE.
14:01
Now where have you heard that before? Well you've heard it everywhere in statistics. You've obviously heard it in standard deviations, you've heard it in ANOVA. Same idea, sum of the squared errors.
14:17
It's a fancy way of saying we add up the squared residuals. And when we do so, it's 120.
14:28
Now when we say squaring the residuals, we literally mean squaring them.
14:35
So, 25 over here in the left-hand side, that's negative five squared. 49 is seven squared, and so forth. Well we actually mean squares,
14:46
so when we square each residual, or error, we're actually making squares.
14:55
So when we say sum of squares, we literally mean the sum of squares.
15:02
So 49 plus 25 plus one plus four plus 16 plus 25 adds up to 120.
15:10
Now, here is sort of the blockbuster bombshell concept of this video.
15:17
The goal of simple linear regression is to create a linear model that minimizes the sum of squares of the residuals, same thing as the sum of squares of the error. So what we're gonna do is we're gonna create a different line through the data, once we introduce an independent variable that will minimize
15:42
the size of these squares. And actually mathematically, we'll come up with the line through the data that minimizes these squares as much as they can be.
15:54
And that will be our best fit line for the data. But again, in this problem, we're only using one variable, we're only using the dependent variable. So when we introduce the independent variable, it will sort of take away for its own self some of this error we see here. If our regression model is significant, it will eat up some of the raw error we had when we assumed, like in this problem, that the independent variable did not even exist.
16:28
So what we're doing here in this problem, we're taking a simple linear regression problem that in theory has a independent variable, called the bill amount, and a dependent variable called the tip amount. But what we're doing is pretending that the bill amount doesn't even exist. We're only using the tip amount. So, that creates a sum of squared residuals of 120.
16:58
Now, when we introduce the independent variable of bill amount, what will happen is that we'll create a different best fit line through our data.
17:12
And what it will do is it will sort of eat up some of this sum of squares. So when we do regression, we're gonna have sum of squares regression and sum of squares error.
17:26
So by introducing that independent variable of bill amount, it will create a new line that goes through the data. That new line will explain some of the sum of squares, and therefore it will reduce the SSE down, the sum of these squares as much as it can be. So the regression line will and should literally fit the data better.
17:51
It will minimize the residuals.
17:56
So when conducting simple linear regression with two variables, we will determine how good that line fits the data by comparing it to this type,
18:08
where we pretend the second variable does not even exist. So when we say a model is good,
18:17
a linear regression model is good, what we're saying is that it reduces the sum of squares of the error by a large amount.
18:28
Which is another way of saying we're comparing the other best fit line to this one you're looking at right here. Simple linear regression is always in comparison to what we would have if we only had the dependent variable.
18:46
So if a two variable regression model looks like this example, what does the other independent variable do to help us explain the dependent variable? Well, it does nothing. If we introduced bill amount into a two variable simple linear regression, but the best fit line looks exactly like this, then the bill amount didn't give us anything. It didn't explain the variability in the tip amount anymore than the tip amount itself did. So we're always comparing our simple linear regression
19:22
best fit line to this one.
19:26
Basically, the mean of the dependent variable alone.
19:34
Okay, so quick review. So simple linear regression is really a comparison of two models. The first one is where the independent variable does not even exist and we just use the mean of the dependent variable, like we did in this video. And the other use uses the best fit regression line, where we go ahead and introduce that second variable, the independent variable in this case, the meal amount, and that creates a different line, and then we compare that to the first one.
20:05
But if there's only one variable like in this example, for this video, the best prediction for other values is the mean of that dependent variable. In this case, it was $10. Now the difference between the best fit line and the observed value is called the residual, or the error.
20:26
The residuals are squared and then added together to generate a sum of squares, literally residuals or errors, or SSE. So we square the residuals and add them together, sum of squared residuals, or most often it's called sum of squares error. So simple linear regression is designed to find the best fitting line through the data that minimizes the SSE,
20:52
that minimizes the area of the sum of squares residuals, that minimizes the area of the sum of squares error. And actually through calculus, it is the best fitting line. Now I'm not going to go into the calculus behind that, but you're just gonna have to start trusting me on faith that when we come up with a best fit line in simple linear regression, that literally is the best fit line that reduces the SSE.
21:23
Okay, so that wraps up our very first video of mini on simple linear regression. So I just want you to realize in this video that later when we talk about the best fit line in regression, we're actually comparing it to the situation where we don't have the independent variable at all. We're just comparing it to the case, where we're looking at the mean of the dependent variable. So in this case, in this video, all we had was the mean of the tips, 10 bucks, that's all we had to go on.
21:53
So therefore, our best guess or best prediction for the next tip was $10. Now later, when we introduced the meal amount, what will happen is we'll get a different best fit line that will explain or take up some of that error in the regression, it'll reduce the error, we'll have a different line, then we'll have smaller, hopefully, residuals. If the regression line is flat across, like we saw in the first example of this one, then the regression doesn't tell us anything,
22:24
the meal amount doesn't mean anything, so the best guess is really just the mean of the tips. So, we'll go more into this example in the second video, just wanted to lay the foundations for that. Look forward to seeing you next time. (gentle acoustic guitar music)
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat