Skip to main content
Loading Explore guides…
Data Science Basics with Python · Acedra
Home
Explore
Computer Science
Data Science Basics with Python
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
150 segments
Copy
Hide video
0:00
can you learn data science even if you don't know anything about data science at all or even python in just an hour you can I'm going to teach you the basics of data science and python along the way and we're going to break down the concepts so that they're very easy to understand and you will actually for the first time in your life understand and use slope intercept form let's go so first off the tools that we're going to use today are visual studio code and something called
0:30
Anaconda Anaconda is a tool that you can install that will make setting up your data science environment a lot easier we have a video to show you how to do that in just a few minutes you can pause take a look at that video and then come right back it's not a complicated process so to get started let's first create a folder where we'll put all of our code so I'm going to create a folder here and let's just call it explore data and then I'm going to open this in Visual Studio
0:55
code and you can do that by just dragging and dropping the folder onto Visual Studio code now we have our empty folder open and we want to write some code now data scientists use a lot of tools but one of the ones they use the most often is something called Jupiter notebooks so if we look in our extensions here we can install jupyter notebooks the other item that we can install if we want to probably better even is just install the python extension for visual studio code
1:25
which gives you intellisense and linting and also jupyter notebooks I already have this installed so let's go ahead and create a new Jupiter notebook so we'll do the new file and call it student data we're going to be working with some student data today exploring students who passed and failed a class and trying to figure out what happened there let's go ahead and create this notebook here and in a notebook we have these things called cells and we can write code in the cells and execute them we
1:54
can also use something called markdown and markdown just lets us annotate student data right this notebook analyzes uh pass fail data for a class okay and then when we hit the little check mark here it just adds that text to the notebook all right so let's go ahead and add a new cell and put some code below so first off we just need some data to work with Okay so let's just bring in a python list which is what we have here and we're going to set it equal to the
2:26
data variable and that variable data now exists but how do we see it well there's a variable Explorer here in Visual Studio code and if we open that and scroll down a little bit here's our list here it has 22 items in it another thing that we can do is that we can just print out from the cell what's in the cell right so we need to say print data and this will print that out below the cell here so these are all of the items that are in the list now lists are great for working with data in
3:01
Python but they're not so great for working with numerical values and a lot of data Sciences statistics and math and so there's a package called numpy that actually has an array type that's better for this so let's add a new cell here and we're going to import numpy as in p okay and then let's create a new variable called grades and then we're going to set that equal to numpy and we're going to create a new array and we're just going to pass in that data object that we have from the first cell
3:34
which will give us an array and then let's just print out that grades object here okay so these two objects look the same what's the difference well the most obvious difference is that there's commas here but not here but the real difference is that the array with this here is much better for doing mathematical operations let me give you an example so up here if we take this data list object and we multiply it by two what do you think will happen let's run and find out
4:06
it actually doubles the number of objects in the array right so here's the first set okay and then here's the second it literally took the array multiplied it by two and gave you twice the objects in the array okay so what happens when we multiply this grades object by two in this case it takes the array and it multiplies each item in the array by two so arrays and numpy are much better for working with numbers for doing mathematical operations on those numbers
4:36
all right so let's go ahead and keep on working with this array object here now one of the things that we can do is we can take a look at the shape of this array so let's do that so we can see that there are 22 items in this array another way that we could see this is if we go back to the variable Explorer here here's our grades array we can open that up and take a look at it and you can see here the same thing 22 items in the array and when the variable Explorer
5:04
launches here we can see all of those items here now notice that each one of these items has an index and this is added for us automatically an arrays start at zero okay so the first item is always in the zero position and then the second item is actually in the first position when you first get started with programming it's a little tough to wrap your head around but you'll get the hang of it very quickly so knowing that let's look at how we can get items out of this array so for
5:33
instance if we wanted the first item in the array because arrays start at zero we would put zero and this would give us the very first item that's this item here okay now if we wanted to get the 47 that would be 0 1 2 right so if we change this to 2 we should get 47 and if we wanted this 3 that should be zero one two three four five this should give us the three okay so what else can we do with arrays well we can do mathematical operations on the whole array so for instance if we wanted to
6:09
figure out what the mean score was okay so the average test score here we could do a mean on the array and we see that it's a 49 must have been a really hard test so what we want to do here is determine why the scores are so low so to do that we need to bring in some additional data into this array and what we want to do is bring in the amount of hours that each student studied along with their grade so let's go ahead and add in this study hours a list item here
6:39
and then what we want to do is just say we have a student data new object called student data and we're going to set that equal to the numpy array okay and then inside of that array we want to have two objects we want to have the study hours which is the number of hours each student studied and the grades okay and what this is going to do is create an array with two arrays inside of it let's go ahead and run this so let's take a look at this in our variable Explorer here so if you pull it
7:08
up you can see we now have two arrays inside and each array has 22 items and if we were to pull that up here in the variable Explorer here's the array one which is the study hours and here's the second array which is the grades now when you're working with an array that has multiple arrays inside of it how do you get a single item from one of those arrays well let's take a look at our student data here and if we pull back the first item the item in the zero
7:36
place we're going to get the first array and if we want the first item from the first array can you guess what we need here another zero so we're gonna get the first array first item from first array so it should be ten so let's run that right if we wanted the first item from the second array that would be one zero okay all right so let's do some analysis here so let's go ahead and do a couple of things first we want to get the average study time so let's go ahead and create a variable
8:09
that's the average study and let's look at the student data right and the study hours was in the zero position the first array so we're going to get the mean of that array of the average and then let's get the average grade okay so that would be student data same thing here and actually in vs code we can just copy this line down with command or alt shift down it gives us the line over and it just makes it easier for us to code here so we can say average grade and then
8:36
we'll just change this to a one and then let's go ahead and print out what this says so in Python that's a print statement and you can use this formatted string so F and then a string and we can say average study time the colon and then we use these little brackets and we can put the variable right in there so average study and then on another line which is backslash n we'll put the average grade same thing average grade and you can see Visual Studio code's helping us
9:08
out here this character here is called a carriage return line break just puts this text on a different line uh whoops okay let's run this and the average study time was 10 hours and the average grades of 49. how'd you like to study for 10 hours and still get a 49. I'd like to know what was on this test when we're dealing with multiple arrays inside of an array it can start to get kind of confusing right just the idea of having a raise inside of an array starts
9:37
to get tough and so there's a package called pandas that will allow us to work with this data in a table which is going to be something that looks a lot more familiar so let's do that so let's add a new cell here and we're going to import a package called pandas now if you're running this and you get an error saying that the package couldn't be found what you can do here in your terminal if you have Anaconda like I do is that you can just issue a conda install command and then install
10:06
the package that you're missing so in this case we could run conda install pandas and that will install pandas so if you run into any of those missing package errors give that a shot let's go ahead and make our data set a little bit more complicated so to do that what we want to do is create this thing called a data frame in pandas a data frame you can think of as a table and we're adding some data here so we're actually putting the name of the student so this is new this is new data and then
10:35
we have the student data right which is our study hours the first array from student data study hours and then the second array from student a to the grade we're putting in grade right the difference here is we're actually naming these things now so we actually know what they are and we can go ahead and look at this thing which is called a data frame and see its structure so we can print it out here from the cell and you see it just looks more like a table
11:01
we're still working with multiple arrays this is a structure that probably just makes more sense to your brain this is called a data frame it's provided by pandas the other way that we could look at this is if we come to the variable Explorer here and we go down and look for our DF students which is our data frame you can see there's 22 elements and it's got three arrays inside of it right but when we open it up it looks more like a table which is a data structure we understand kind of like a
11:30
spreadsheet all right so how do we work with data frames well let's take a look at how we can get an item out of a data frame so in a data frame let's say we wanted to get the first item Now with an array we would just use zero right so we wanted the first item from the first array it would be zero and one if we run this on the data frame though we get an error because data frames don't work that way what we need to do instead is we need to use the Loc method and then we can pass in and say
11:59
give us the first item here and the first item is it's really a row right so it's Dan Dan's study hours in dance grade it's that first row and if we were to go back and look at the data frame right here so if we wanted for example Vicky in the fifth position here we could go ahead and just pass in a five and that will give us actually uh Vicky who only studied for an hour and got a three ouch now let's say that we only wanted to know the grade from Vicky right so we
12:31
want the item in the fifth position but we only want the grade so we can actually pass in here as the second parameter the column that we want back and because the data frame knows about the column names we can just tell it the grade column and if we run this we just get back the 3.0 and if we change this to name we should get back just Vicky so there's a lot of other ways to get stuff out of data frames and searching for data so one of the ways is to use filter
12:56
Expressions so we can take this data frame of students and we can say for the name column give me the data where the name equals Vicky that's what this expression says here so let's run this and we get Vicky and her study hours and terrible terrible grade now another thing that we could do here instead of doing it this way is we can use something called the query method so we can write query here and then we can actually write a query which is a string but it is essentially the same thing we
13:26
just say where name equals Vicky but Vicky has to be in quotes because Vicky is a string data type so if we run this we should get back exactly what we had before but we don't because I left out an equals the double equals is important and now we should get back Vicky there we go all right another thing that we can do here is if we go back to the way we were doing it before where we're saying DF students we can actually just hit the dot here and then the data frame
13:51
knows about the columns that it has so it knows that there's a grade column here right or it knows that there's a name column and then we just do equals and say Vicky and run this and we'll get back Vicky and this is the method that we'll be using for sort of the rest of this video to access data in a data frame all right so those are the fundamentals of a data frame sort of back up and take a look at a more real world scenario something you might actually do so it's common in data
14:19
science to import data from some sort of an external Source in this case we're pulling in a CSV file so we're downloading a CSV and we're loading that CSV into a panda's data frame so we're reading in the file the delimiter is a comma and we're inferring the column headers let's just run this so that we can see how it works so if we run this here and then look at our file explorer you'll see that the CSV file actually gets downloaded right and here's the CSV file here's the name the
14:52
study hours and the grade it's kind of like a table in comma separated format so what we have here and let's go ahead and clear the cell output to make this a little less messy and if we look at this data frame again this DF students it's going to look exactly like it did before only this time the data has been loaded from this CSV file instead of us just loading static data because usually data scientists are working with lots and lots and lots of rows thousands of rows
15:18
but this data is a little bit different and the difference is that it's a little bit more real world and one of the ways that it's real world is that there are some null values somewhere in here okay so one of the first things that data scientists have to do is they have to clean up the data so that they can actually work with it so in this case to find any missing values we can say DF students and then just look for is null right so if we run this it will tell us
15:45
whether or not the value in each column is null or not null so in this case false it's not null in any of these places but this isn't super helpful for us there is a better way for us to see exactly which values are null so first of all we can hear and just say sum and just get a sum total of how many uh values we have that are null in each column so there's one value that's null in study hours and two values that are null in the grade column so the next question is then well what do we do
16:20
about that well there's a couple different things that you can do one of the things that we could do here is we could go ahead and replace anything any missing value with an average value right so for missing a grade for student one of the things we could do is just replace that with the average of all the other scores since we don't have a score for them so what we can do here is we can fix just the study hours column to start with so let's go with DF students and then study hours so
16:47
we're going to set that equal to and then we're going to fix the missing values inside of that column so we're going to say DF students dot study hours dot fill in a and basically we're telling it we're going to fill in all of the missing values and then in the parentheses we need to tell it what we're going to fill it with so we're going to fill it with the study hours and we're just going to calculate the mean of all the study hours and we're going to fill any missing study
17:14
hours values in that data frame with the average all right so if we go back and run this we should see that study hours now shows as having zero null values right because we filled in the one missing value with the average the problem with that is that it kind of makes your data not correct okay and the reason for that is because you don't know if the student got the average they could have gotten less than Vicki right and pulled the whole average way down and so sometimes
17:41
it's better to just delete any null values in a data set which is commonly what a data scientist will do so let's go ahead and do that so we're going to set our whole data frame here and we're going to set it equal to DF students and then we're just going to drop we'll use the drop in a to drop any rows that have null values so to do that we need to tell it which axis we're looking at rows our axis zero columns are axis one right so if we set axis one it would drop
18:14
columns we don't want to drop columns we want to drop rows so we say axis zero so if we do that and we run this and then we go back and look now we'll see that we don't have any null values right we've just dropped the rows that had null values in them all right so let's go back to where we were before when we were working with our arrays when we calculated the mean study time and the mean grade and we just printed that out this is that same thing just with a data
18:42
frame so the average study hour is 10 and a half average grade 49.18 so let's go ahead and get all of the students who studied for more than the mean amount of study time so to do that we'll take our data frame of students and we'll say give us everywhere where the study hours was greater than the mean study time okay let's go ahead and run this and now we get everybody who studied for more than the mean amount of time which I believe was 10.52 so this is everything that's more than 10.52 so it
19:17
would be interesting to know here for these people who studied more than the average what was their average grade well since this actually returns another data frame as you can see here we can just say for this grade column give us the mean right what is the mean grade for everyone who studied more than the average amount of hours if we run this we see 66.7 if we assume that a 60 is passing and the mean is 66.7 it would be nice if we could go ahead and flag in the table who passed
19:51
and who failed so we want to add a column to this data frame which just says whether or not someone passed so to do that let's create a variable called past it's going to be our new column and then we want to use this thing called series on PD and you see the little yellow squiggly it's because we haven't imported pandas we did but then we deleted that cell so let's go ahead and put it back here
20:17
there we go and in that series what we want to do is just say from the data frame here with the grade was the grade greater than or equal to 60 right this return to true if it was and a false if it wasn't and it puts those trues and falses in the past variable and then we want to take that data frame and we want to concatenate here this past column onto the rest of the data frame adding a new column so we're going to say we want to take the students here and we want the past
20:50
column and then we can take this opportunity to rename it and just make it say passed here with a capital letter and then finally we need to tell it what axis are we trying to append a row are we trying to append a column remember rows are axis zero columns are axis one so we'll say axis equals one so this should append a new column to our data frame so let's run this and then let's go ahead and come back to our data frame here and refresh and look we have a new column with a false or true about
21:21
whether or not someone passed or failed the test now because data frames are designed for tabular data or data that's in tables we can perform all sorts of operations on it for instance we could put students into two groups those who failed and those who passed and look at the data that way and that's called a group I so let's do that so in our data frame here we're just say Group by and then what do we want to group by well we want to group by the students and we have that new past
21:51
column that we added very handy and then we want to get the count of how many passed or failed in each group so if we run this it tells us who passed how many and how many failed but it does it in the table format which is kind of hard to look at so if instead we just take the name right if we Group by just one dimension the name and we run that then we'll actually get just one set of numbers here 15 people failed and seven past crazy hard test now what if we wanted to know for all of
22:26
these students what was the average study hours in grade for those who passed and those who failed to do that we can just print out our data frame here of students and use the group by and we want to group by that past column and then we want to know though from that we want to know the mean of the average study hours and grade or the average study hours on grade so we could look at the grade and then say mean like this but this will only give us the mean if we want to get the study hours
23:02
and the grade and the mean of both of those so to do that we want to take our data frame of students we want to group by and we want to group by whether or not those students passed but we want the mean of both the study hours and the grade hmm well because we know that this is actually data frame itself when we group it we could say grade and then give us the mean and if we run this it actually will give us that but we if we want the average study hours and the grade how do
23:33
we do that we can actually do both of those at the same time using this array syntax here and saying give us the study hours and give us the grade and calculate the mean of both of those so if we run this
23:50
you can see that we now get the average study hours for those who failed eight and those who passed 14 and the same with the grade so we can do both of those things at the same time using that syntax so the last thing I want to show you here is that data frames are really versatile and you can do a lot with them uh including like sorting the data and descending your ascending order now these data frames that we've been looking at they're really versatile but
24:16
one of the things you might have noticed is that when you alter a data frame you have to assign the value back to the original data frame if you make a change to the data frame and you don't assign it back to itself it doesn't actually affect or change the data frame so what we can do is come in and say our data frame and then we set it equal to itself we're going to sort it let's say we sort it let's say we want to sort it by grade and we want to sort it just ascending by
24:41
grade so let's go ahead and run this and then let's come back to our data viewer and refresh and we should see yep Vicky and her terrible grade there at the bottom sorted by grade if we wanted to sort by name or study hours let's say study hours and then if we wanted to do it in reverse right we wanted to say highest to lowest highest first then we would say ascending is equal to false and if we run that come back to our data frame we should see study hours the most of the top descending here
25:14
and we do descending something that data scientists do a lot is work with charts and graphs and this is because when you're working and trying to identify complex patterns in data it's easy to see if you can look at it it's hard to see if you have to examine the numbers themselves so charts and graphs help the data scientists do that and we're going to take a look at how they do that now so the first thing we're going to do is we're going to start with the data set
25:40
which is exactly the data set that we were exploring before with student data we'll go ahead and run this and this will pull in the grades CSV file which we already have but it'll get pulled in again because we've made additional requests for it see right there grade csv.1 and then here we're pulling in the grades putting that in a data frame like we talked about adding that passes column whether or not they passed or failed and then just printing out the data frame here so it's exactly what we
26:08
had before when we were exploring the data and learning about data frames by the way when I jump over to these CSV files here you might notice that they're different colors and this is powered by an extension called rainbow CSV you might want to check that out right here so one of the cool things about Visual Studio code is extensions and there are tons of extensions for those of you doing data science working with data one of those being rainbow CSC all right so we've got our student data
26:33
now we want to make a chart or a graph so to do that we're going to use a library called matplotlib and again if you don't have this installed you can just in your terminal run conda install matplotlib just like this and that will install it so if you run this and you get an error try installing matplotlib in either python using pip or if you're using Anaconda like I am with conda install so first let's pull in matplotlib here and then we're going to import Pi plot which is the plot plots
27:04
are the charts as PLT and that's what we'll call it here the variable PLT
27:13
now what we want to do is create a bar chart here to look at this student data and along the x-axis we're going to put the students name okay so name across the bottom and then we want their grade across the side so you would think that this the the y-axis would be called y but it's not when you're using matplotlib the the plot item here it's called height I'm sorry I didn't name it and here we're going to put the grade for the students and with that we can
27:48
just go ahead and say plt.show and this should be all we need to create a bar chart of the students and their grades and there we go we have a bar chart it's not easy to look at but there it is so let's take a minute and customize this chart make it a little easier on the eyes so let's start customizing here let's add a title to this chart which is called student data student I'll tell you what let's talk student grades or descriptive there and then let's add a
28:15
label to the x-axis which is just X label we'll call this name and then the Y which was height before is actually now y label it is y label and we'll call that grade and then the most problematic thing with this chart is that these X ticks are all running together and you can't read them so if we turn them 90 degrees they'll be a lot easier to read so let's go ahead and do that let's say plot dot X ticks and then we're just going to say rotation equals 90. so
28:42
we're going to rotate them 90 degrees just go ahead and run this and now this chart's a whole lot easier to read because we can actually read the names on the bottom and there's our favorite student Vicky with that horrible score so we can also do things like change the color here so if we change the background say well excuse me not the background but the color of the bars this will change the color of the bars to Orange so we have a lot of control over the charts in the way that they
29:06
look when you're working with charts what we've got here is something that we've called a plot but it's important to be specific plot is actually the chart that's the plot the whole thing itself if we include all the way out to the edge that's called a figure now the figure can have multiple plots or charts within it and the figure is created for you automatically if you don't specify it but if you do specify it you get a lot more control over the size and shape
29:31
of the chart so up here at the top we're just going to Define a variable called Figure and then we're going to say that that is a plot figure and then once we do that we can start to change the dimensions of the plot itself and we can do that with figs so we might want this chart to be wider or taller just changing the dimensions now the Fig size takes two variables a width and a height but they're in inches so for instance if we wanted a chart that was 10 inches by
29:56
4 inches 10 4. but because we're dealing with screens we don't think in inches we think in pixels so one thing that you can do here is because the DPI is 100 by default on these plots in matplotlib you could just multiply by a hundred to get the pixel so this would be a thousand pixels by 400 pixels if we wanted something that was 1280 by 720 that would be 12.8 by that's right 7.2 and we actually don't need to change anything else we just go ahead and run this it will
30:27
resize that chart to that size so that's an important thing to know is just how you make charts bigger or smaller with matplotlib and you do that with the figure and fix size okay now as we said this whole thing is called a figure and a figure can have multiple plots those are charts and that's commonly done because it helps us visualize data and we're going to do that now when we want to have multiple plots in a figure we need to use the subplots item on the plot element there and then
30:56
where the fix size is the first thing we need to tell it is how many rows and how many columns this figure is going to have so in this case we want to have one row and two columns and the fixed size is a little bit big here let's take this back down to ten by four or a thousand pixels by 400 pixels when you don't change the the standard DPI now we no longer are working directly with the plot element and instead we're working with a an axis element all right so that
31:24
actually is returned by the subplots here along with our figure and we'll just say on the first axis okay we have two columns 0 and 1. so we're going to put a chart in the First Column which is 8x0 a chart in the second column will be ax one so in ax0 we're going to put a bar chart and the rest of this pretty much stays the same now we're no longer working with the plot element and instead we're working with this ax 0 right that's where we're defining the plot and instead of title it's set title
31:53
and instead of all this stuff here we're just gonna have this 8x0 like we did before down here and instead of X text we're going to say set X tick labels and then inside of that we're going to say DF students do you have students.name and they're going to rotate the labels 90 degrees just like we did before and this gives us basically the same bar chart that we created now we want to create a pie chart to show the pass and fail totals we're going to put that next to the bar
32:23
chart so to do that we're going to get a total count of all of the passes so a count of how many people failed and how many people passed so to do that we're going to take our data frame and we're going to look at the past column and in order to sum up all of the people who passed all the trues and all the falses what we need to do is use the value counts method here which will give us some of all the truths and the sum of all the falses two values and then we
32:50
want to pass those in and make a pie chart so on the axis one which is the second column we're going to create a pie chart and in that pie chart we're going to put the past counts as the data that we want graphed and then for the labels we're just going to use the pass counts as well because that's going to tell us true or false false and that's all we need to know here and then just like with the bar chart if we want a title we can call set title here and pass in a new title let's call this
33:17
passing grades perfect and then finally we're just going to call fig.show right because we don't have a plot element anymore we have a figure element okay so let's go ahead and run this and see what we've got
33:31
all right we got an error we did get charts but one of them didn't render and we got an error but this is great because it gives us a chance to debug us and see what went wrong so if we come up here to the top here we can see this which is called a stack trace and we can see it's telling us line 11 here this line This label right here it's telling us got an unexpected argument label okay and that's because it's labels not label right and this is what makes programming
33:57
kind of tedious sometimes it's because you have to get this exactly right so let's change that run it and now look we have our passing grades on the right as a pie chart and our student grades on the left two plots one figure all right now it's worth noting that charting and plotting is so foundational to data science that you don't even need to use matplotlib directly it's kind of built into the data frame itself so we can actually plot this or chart this directly from the data frame so let's
34:26
call DF students and then let's just say that we want to do a plot what kind of plot well we want a bar plot on the x-axis we want the name and in this case it is the why not the height this time which is nice so on the Y we want what we want the grade and then we can actually set the big size here as well so let's set that equal to 10 by 4 seem to be a good one and then just like with the previous chart we can set the color let's set it to something else here this
34:52
is actually all you need to create a nice bar chart really easy to do directly off the data frame all right now it's time to get into some actual data science here and figure out what's going on in this data and what what we want to do is take a look at these grades and something that data scientists are often interested in when looking at data is the distribution of that data across the sample now what's a sample the sample is the data that we have okay the grades the grades here are
35:21
our sample and then there's the population the population could be any student who might ever take this class so we look at data for the sample because we can't look at the entire population and then we make an assessment or a judgment about the entire population based on the sample we have the larger the sample and the more data points the better our prediction can be for the entire population when we examine the distribution or how the data is spread out across the sample a great
35:49
chart for doing that is called a histogram so in our notebook here let's create a histogram so let's get some data here we'll just call it grades and then we'll just pone in from our dsstudents.grade column and then let's create a new figure and then we're going to just say that we want to plot with a new figure and this allows us to set the Fig size right and then we'll just make that a 10 by 4 I think is what we've been doing so far and then we're going to use the plot
36:16
item to plot a histogram we're just going to tell it that we want a histogram and we want the histogram to be the grades the data that's in the grades variable there now just like we did before we can customize this let's add a title let's just call it data distribution and then let's go ahead and set our X label here we go and we'll just say that this is the value and then we'll set our y label and this is the frequency at which this occurs and then let's just go
36:46
ahead and show this figure and run this and we get a histogram so let's talk about what we're looking at here the histogram shows the distribution of the data again across the entire sample so from 0 to 100 this is where the data Falls you can see that most of the grades are actually right between about what 30 40 the low 40s and 60 right maybe mid 40s and 60 right there so most people scored between about a 45 and a 60 and that's why that's the tallest point and then over on the very right
37:21
you have the people that scored the highest grades which is only a few of those then over on the left you have Vicky who's bringing the whole average down all right so this histogram is a symmetric shape with most of the data being in the middle and and the fewer being out on the edges which is quite common when looking at sample data for a population so another thing that data scientists do when looking at a histogram like this is try to Define What's called the central tendency and
37:46
that basically means what's the middle point of the data now there's a couple different ways that you can measure this one of them is something called the mean which is an average another one is called the median which is the number that is right in the center if you line up all of the data from lowest to highest what's the number that's right in the middle equally from both ends and last is the mode and the mode is the number that occurs most often in a sample so let's
38:12
go ahead and add that to our chart here and see if it tells us anything different all right so right here before we actually plot our figure let's go ahead and get the Min value which is our students.grade dot men so next we're going to look for the max vowel we're going to set that equal to dfstudents.grade.max and as you can see this these lines are starting to look very similar and we actually need the mean the median and the mode so in vs code we can use a shortcut here to do
38:40
that where we copy down lines and we do that using option shift down and that just copies the line down we can do that two more times here right so here let's set a Max Val we want let's say the mean and then let's come out here to the edge and get the mean and then for this next one let's do the median I actually just call it there we go and then pull in the median and all of these are built-in functions right on the data frame which is really nice and then the
39:10
last one is the mode and as you can guess come out here and ask for the mode the one difference with the mode is that you might have multiple numbers that occur the same amount of times right so maybe the score that occurs the most is 65 and then the score 68 also occurs that same number of times and so because of that mode is a list of items so we need to make sure we get just the first item just by asking for that zero placed item there all right so let's go ahead
39:39
and print these values out so we can actually look at them and then we want to add this to our chart so to plot lines on a histogram you use the axv line method here and this adds a line we need to tell it the line the value that we want to add here is the Min value and then we can do things like specify the color okay now because we're going to need a line for min max mean median mode again we can just use the option shift down or alt shift down on Windows key to
40:09
copy lines down just like this right so we can make this one this is the Min we can make this the max valve it can also be gray let's make this one the mean vowel and let's make it uh cyan and oops I capitalized a v here and we get that nice squiggly to tell us that it doesn't recognize if you see that see that squiggly if we actually Mouse over it'll tell us it's not defined which means we use the wrong variable name so let's go ahead and correct that and then we can make this one the median
40:40
value and let's just make that red and then lastly let's take this and make this the mode value and let's make that yellow should be fine now that we've done all of that let's go ahead and run that again and check out our chart so now we have a histogram and we have our lines here so if you remember the gray line is the Min this gray line is the max this is the mean median and mode and in our sample they all happen to be in the middle right we could pretty much
41:06
tell that by looking at the histogram some histograms aren't this neat it's not as easy to see so it's helpful to chart those lines in our case they are all right smack dab in the middle that is the central tendency of this sample now another chart that data scientists use a lot to examine central tendency is something called a box plot so let's make one of those so let's go ahead and Define that grades variable again we actually don't need to do this because
41:32
it's already in the notebook and it will remember from previous cells but let's go ahead and do it just so that we don't get confused about what data that we're looking at here on the grade and then let's go ahead and create a new chart here a figure we'll say the Fig size is 10 by 4 just like we've been doing and then let's go ahead and add a box plot just called box plot there and we want to pass in that grades data perfect and we can set the title oops title there we
42:01
go just call this data distribution and then we'll go ahead and show the figure all right so what does a box plot look like let's find out this is a box plot what a box plot does is it shows you how the quartiles of the data are split up what does that mean it means that if you separated the data into four equal groups where would they be in this chart the middle two quartiles right so 50 of the data is actually here between about 36 and lower 60s I can't tell exactly 50 of the data
42:38
is here 25 is here 25 is here so this is called a box plot these are often called whiskers and this is something else data scientists use to measure central tendency it's often useful to combine histograms and box plots together to get a clear sense of the data so to do that I'm just going to copy in a bunch of code here and this is code we've already written right men Max mean mode and then we're using the subplots so zero is our histogram with our lines on it and then
43:09
one is the box plot here and then we're just showing that that's it we're not doing anything special here I just want you to meet and see me write that code all over again I'm gonna remove this for a second and run this and then we'll see that we have both of those charts now in the same figure but the box plot doesn't help us much because this chart goes this way and this chart goes this way and that's why I removed this value here up here where it says box plot so what
43:35
we can do is just say vert and just set that to false let's run this again and this will display the box plot horizontally just like the histogram which helps us understand it a little bit better and again we can see here that it matches right central tendency is Right smack dab in the middle fifty percent of our grades being right around here just a different way to visualize That central tendency now we talked about samples and populations and how a sample is a subset of the population
44:02
into the data that we have and we try to extract insights about that sample so we can make predictions about the population if we have enough samples then we can calculate something called a probability density function that will let us make sort of assumptions about the population at large based on the sample the pi plot which we've been working with in matplotlib provides a very handy plot to do just this so let's create a function to do this actually when you have a lot of code it's very
44:30
nice to put it inside of a function and the function just sort of encapsulates all the code that goes together and you can call that function later on so here let's go ahead and create a new figure and we'll say plot figure and then fig size you should be an expert by now at these figures and then we'll go ahead and say that for this data here that we've passed in we want to plot and we want the density and remember we can call the plot directly on the data because it's a data frame and data
45:00
frames have that charting built in and then we can set some additional things here like we could add the title so we'll just call this data density there we go and then lastly we just need to show this thing and then how do we call this function well we just sort of call it so we'll just click show density and then what data do we want to show up on the plot the grade so let's call it DF students and the grade and we'll pass that in as the data so let's go ahead and run this
45:30
and this is our probability density also known as a bell curve you've probably seen these before so you can see that the grades are distributed right here right with the outliers on the edge it looks just like the histogram so from this we could make some sort of assumptions about the data or the grade the class the test we could say the test was too hard if more people got a 50 that got say a 75 and maybe we want to move this over to the right so we want to grade on a curve but that's not
46:01
really the kind of conclusion that data scientists are often after you don't need charts and graphs to determine if you need to grade on a curve teachers do this all the time what a data scientist would do is actually look at how the different points in the data relate to each other and figure out if there are some predictions that we can make about the population based on the sample and that's what we're going to do so first let's just create a new notebook here
46:24
and we'll just call it real world because we're going to do some real world analysis here and then let's go ahead and start with this distribution plot that we had so go ahead and run this this code should make sense to you here let's scroll down and what we'll see here is our data frame and then below that we'll see the histogram that we had and the box plot right with the mean median and mode all right in the center this should look very familiar to you okay so now what we want to do is
46:47
look at a different set of data because we've been looking at grade so let's look at study hours so to do that let's call our show distribution function and this time we'll pass in the data frame of students and the study hours instead of the grade we've got a capital T which is wrong there let's go ahead and run this and let's look at the histogram and box plot for this one all right so this histogram and box plot looks quite a bit different so you can see that the whiskers on this
47:12
plot start at about six and that's zero over to their left here on the bottom indicates that we have statistical outliers less than six but they're outliers and an outlier means that it's not representative of the normal sample here and that could be for a variety of reasons so if you had a one here one hour maybe that's a typo they meant to put 10 hours in the data set and they got one or maybe there's something else that caused the student to only study for one hour we don't know but we can
47:42
safely say that this is a statistical outlier and it's not representative of the rest of the data in our sample so one of the things that we can do here is we can just remove it so let's go ahead and take our students right and then from that let's pass a filter you should remember this from the first video and we're just going to take it where the study hours is greater than one and then we want from that the study hours and this will give us the study hours that are greater than one and then
48:11
let's go ahead and do our distribution again and we'll pass in that new filtered set of study hours and now our box plot goes all the way to the left and the right extends the whole range because we've gotten rid of that statistical outlier now normally in data science you don't want to do this you don't want to just remove statistical outliers without being able to explain why they are there because the thing is if we expanded our sample to the entire population we might find out that a lot
48:38
of people only study for one hour and it needs to be in there so just something to be aware of so now let's look at the density for this distribution so again this is code that we wrote in the last video This should look familiar to you if we run this what we'll see is that bell curve that we generated before or but this time for the study hours and this bell curve looks quite a bit different this bell curve is what we call right skewed and that means that most of the data is on the left side
49:06
right or less than 10 study hours and so it causes the right side of the chart to be longer to have a long tail and those three lines in the middle are the mean median and mode there which you can see are all sort of right around the middle point so we've talked about the importance of knowing where the middle point is in the data but there are some other measurements that data scientists use to gain insights about the data these can be the range or the difference
49:31
between the minimum value and the maximum value something called the variance and something called the standard deviation let's talk about those in statistics variance is described as the average of the squared differences from the mean what does that mean well let's calculate it and it'll actually make a lot more sense it's not that complicated so what variances is it's sort of a measurement of how far the data is spread out across the sample or how far away it is from the mean the
50:02
mean being the average so first let's just get the mean here we know how to do that right so let's get our students our study hours here and then we'll just get the mean from our study hours now let's go ahead and calculate our variance we're just going to start at zero and we'll we'll calculate it and then what we're going to do is we're going to Loop over all of the study hours and we can do that here just like this with this for Loop and this will Loop over all of
50:27
the hours values in the study hours column let's just go right through that whole column and then once we're in there what we need to do is we need to subtract hours from the mean okay so for each row we're going to subtract the hours from the mean some of those values are going to be negative right because if the value is above the mean or the average if the average is 10 and you studied 15 that's a minus 5. so to account for this what they do is they square that number
50:58
which always gives us a positive number and then what they do is they take the variance they add all those numbers together to get uh what they need first off for the variance so we're going to say variance equals and then we're going to just add 2 what the variance already is and we're going to say plus and then order of operations put all the stuff in parentheses so it does the right side first and then the last thing that we have to do to get the variance so variance is
51:24
actually equal to variance divided by so this number we're calculating in the for Loop divided by the total number of uh rows that we that we had in the loop so we can get that by doing DF students and if you remember before there's a shape method that we can call here that gives us the rows and the columns and so if we just do 0 that gives us the number of rows right and so then let's just look at the variance value here so we'll run this and we get 11.607380 what does this mean actually
51:57
it doesn't mean anything to us variance is a measurement that often means something to statisticians but when we're working with the data it doesn't mean a lot to us and that's because we squared those numbers so what statisticians do is they use a different measurement called the standard deviation you've probably heard of the standard deviation before and the standard deviation is just the square root of that variance so to get the square root let's just import the math
52:25
package here and then we'll just ask the math package for the square root function of our variance right and that should give us the standard deviation so let's run this and there we go 3.4 now what this means is that for one standard deviation the value is going to be 3.4 away from the mean either above it or below it but what does that mean well there's a number that you need to memorize here and it's 68 just remember that number in your brain 68 is the number that statisticians use for a
53:04
normal distribution right when that curve looks relatively normal we can say that 68 of the values in that curve will fall in one standard deviation so 68 of the hours the study hours are going to be 3.4 above the mean or 3.4 below the mean and that's the standard deviation fortunately we don't actually have to calculate these things by hand even though we just did so instead what we can do is let's Loop over our columns here in so we're going to Loop over the grade column and the study
53:41
hours column and that's what this for Loop is going to do and then inside of that oops Visual Studio code telling me that's wrong let's fix that and inside of that um we're just going to go ahead and get the column from the data frame here it's going to be either grade or study hours this allows us to do that and then let's calculate the range right which we said was the max minus the min okay and then let's go ahead and calculate the variance here which we can
54:11
just do like this right on the column and then we get the standard deviation the exact same way and then let's go ahead and print out uh what that standard deviation is here and the range and the variance let's run this okay so we can see for the grade here that the range is 94 and the standard deviation is 21 and what that means is that we can do this we can say we can just let's just print this out so we can actually look at it but we can say 68 right that's the magic number you were
54:40
going to store in your brain of grades will fall between right and then we need the mean minus the standard deviation so let's just say uh df.students.grade dot mean minus dfstudents.grade dot standard deviation right that gives us the lower end and then the upper end would be this same thing here let's go ahead and just copy this over okay put this at the end and then change this minus just reverse this sine and then we can actually look at this and it should make more sense so 68 of all
55:23
grades will fall between 27 and 70. okay and that's one standard deviation and these calculations are actually all built into the data frame so we can just describe the data frame here like this as a method and that will tell us for each of these columns what the mean is what the standard deviation is what the quartiles are the Min and the max okay so if you've made it this far you now should understand Concepts about the distribution of data variance and standard deviation range mean all of
55:54
that stuff congratulations now that we know that we can use that to examine relationships between variables in our data set and this is where things start to get really interesting so what we can do is Define a new data frame here called dfsample and we're just going to set that equal to DF students and let's pull out where the study hours is less than one okay and then let's go ahead and look at this data frame in our data frame we have a couple of different kinds of values okay we have
56:25
numeric values and we have categorical values the numeric values are study hours and grade and the categorical values are name and pass we can actually compare numeric columns to categorical columns to get some insight and I'm going to show you how to do that so first off let's compare the study hours numerical value to the Past categorical value so to make this comparison between these two values let's go ahead and use a box plot so let's go ahead and use our
56:56
sample data frame and we can call the box plot right off of the data frame and then we need to tell it what column is it that we want as a bar and we're going to say study hours and it's got this nice little by a value that we can pass here and we're going to say give it to us by pass and I'll show you what that does here in a second and we'll set our fixed size equal to eight by five which would be what 800 by that's right 500 pixels let's go ahead and run this and
57:23
here's our box plot now you can tell just by looking at this that the more that you study has a direct correlation with whether or not you passed the class or the exam it's just readily Apparent from looking at it and probably common sense you could have guessed that without a chart but the real insight here is that these study hours might be a good predictive measurement for whether or not a student is going to pass the class now let's examine our numerical values
57:56
and compare those so let's take our sample data frame here and let's make a plot and along the x-axis the bottom we want to have the name of the student right and you should remember this from the box plot we made in the previous video along the y-axis I want to have two bars we're going to have two bars for every student one bar that represents their study hours and one bar for the grade so to get two columns into the y-axis we just need this array syntax here and then we'll tell it give
58:25
us the grade and give us the study hours okay and then the kind of chart we want here is going to be a bar and let's go ahead and set our fig size here equal to uh let's just keep in using eight by five Perfect all right let's take a look at this okay so we have a bar chart here but it's kind of meaningless and the reason it's kind of meaningless is that the scales for these numeric values are different a grade can be from zero to a hundred whereas study hours don't go
58:58
that high right I mean study hours might be 20 is is our time in fact we can go back up here and look and see when we described our data frame what was the max amount of study 16 okay and the max grade is 97. so they're not on the same scale so we can't actually compare these values like this so what we need to do instead is normalize the range for these values and there's a lot of different ways to do that we're going to use one called a min max scalar and what the min
59:25
max scalar does is it takes the range and it just normalizes it to between zero and one zero being the bottom one being the Max and it normalizes both the grades and the study hours so we can actually compare those two so let's go ahead and create a new cell here and we're going to use this scikit learn Dot pre-processing here and then we're going to import this min max scalar there it is all right so let's go ahead and create the scalar and we want a min max scalar and then we're
59:57
going to take and create a new data frame which we're going to call DF normalized because we've normalized this data and we're going to take our sample data here okay and then what we want to do is we want to pull off the name grade and study hours columns because that's what we're using in our bar chart any time that you specify multiple columns that you want from one data frame to another you got to use this kind of goofy double bracket syntax all right so
1:00:23
we want the name we've got the grade and we want the study hours okay and then what we're going to do is we're just going to call Copy which is going to copy all that into the new data frame it's a different way to create a data frame from another data frame we're just selecting specific columns and copying the data over now let's go ahead and normalize this data frame so let's say DF normalized and then let's tell it what columns we want to normalize and remember when you're specifying multiple
1:00:49
columns in the data frame you've got to have that goofy syntax there so we want the grade and we want the study hours okay and then what we want to do is we want to tell it that these columns are equal to and let's call that scalar that min max scalar and we're going to say fit transform okay because we're using the min max scalar and then there we're going to pass in the normalized data set here with that goofy syntax so normalize the grade and the study hours
1:01:19
okay and then the last thing we want to do is plot this out so let's take our normalized data frame call the plot here X's name this is exactly what we were doing before y since we want both grade and study hours we need the Bracken syntax here study hours the kind is going to be bar and the fixed size we will make uh eight by five okay let's go ahead and run this
1:01:49
and now we've got a bar chart where we can actually compare these values and again you can tell just by looking at the chart that the number of study hours is directly correlated to the grade I mean you can tell that just by looking the more hours you study the higher the grade tends to be if you study for just a few hours or like this person here the grade is really really low so there seems to be a correlation between study time and grade and there is a statistical correlation value that we
1:02:20
can get that will tell us the strength of the relationship between the number of hours that you studied and the grade that you got so to do that let's create a new cell here and let's take our normalized data frame let's get the grade column here and let's ask for the strength of the correlation to this other column here which is study hours okay and then we'll go ahead and print that out that's a 0.91 or 91 correlation which is super high the correlation value goes
1:02:51
from -1 which means they're probably not at all related to one which means statistically the probability is they are highly correlated correlation goes from a value of -1 to 1. minus one being they're probably not related at all and a one being that there's a really good chance that these two things are related and that 0.91 is almost a one that's a very high correlation now you may have heard the phrase correlation is not causation and this is really important
1:03:21
to understand because it's going to be super tempting for us to jump to the conclusion here that the more that you study the higher the grade that you're going to get that they're directly related and even though the causation is high that's not necessarily true and we can actually see that just by going the other way and saying well because of the strength of the correlation they studied a lot because their grade was high which doesn't make any sense at all so let's
1:03:45
continue to examine this relationship here let's go ahead and take our sample data frame and let's create a new kind of plot here let's create a scatter plot let's set the title here to study time study time if we can spell versus grade and on the x-axis let's put the study hours and on the y-axis we'll put the grade all right let's go ahead and take a look at this scatter chart now we can put a line here to better see the relationship between the dots the grade
1:04:15
and the study hours that shows the trend if we use something called a regression line now stop for a minute do you remember when you were in high school and you took algebra and you learned about slope intercept form remember that equation y equals MX plus b and you thought when am I ever going to use this well hold on to something because we're about to use it so our formula for slope intercept form was y equals MX plus b please don't turn the video off this is
1:04:48
going to make plenty of sense here in a second so in this equation here Y and X are the coordinate variables we actually already know what the X and Y values are right access study hours and Y is great what we need to do is calculate the slope of the line which is m and b which is where it crosses over the y-axis if we have all those values we can calculate a new y point for the line based on each of our X values and we actually call that variable f x because it's actually going to be a function
1:05:19
that we pass these values into it's going to give us back a new position so that we can draw a line of best fit of the line that's best fit for the points we also have to deal with something called an error and the error is the difference between the actual y value and the new y value that we calculated for the line the regression line that we're going to draw and the way that we calculate the error is that we actually take and we find the difference between the actual y value and our new
1:05:52
calculated y value and then we Square it and remember we Square it because sometimes it will be negative and this gives us a positive of number and this line gives us a line of best fit and it's actually called a least squares regression and the reason why is because we're trying to minimize the error the the air is squared and the line is a regression line it's still with me now fortunately you don't have to calculate any of this stuff yourself you just have
1:06:17
to conceptually know that it exists and how to use it to draw that line python will do the rest for you so let's go ahead and use this SCI Pi package here and import the stats from that and then we will want to do is let's create a new data frame called DF regression and from that let's get our sample and then we want to pull out the grade and the study hours so let's pull in the study hours here and the grade and then we'll just copy those in that's that goofy double
1:06:44
bracket syntax we talked about and then we want to calculate the slope M and The Intercept B okay so we can do that by calling this stats dot stats Dot line regress here okay this gives us the regression line and we just need to tell it well what line are you trying to calculate we need the X and Y values what is the x value so dear progression study hours right is our x value here and then the Y value study hours and then the Y value is going to be the grade okay and this will give us the
1:07:24
slope and The Intercept now unfortunately it's going to give us a bunch of other variables as well and we can actually see this if we Mouse over it's going to give us the slope The Intercept the R value the p-value and the standard error so we can just go ahead because we need those variables or this function will fail we'll get the r value the p-value and the standard error even though we're not going to use those we have to include them for this code to actually work all right so now on our
1:07:53
data frame we can have a new column that's going to be that new y value and we can say that's equal to y equals MX right M times our regression of study hours because that's the X study hours what plus b right and that's going to be just out here plus b because we already have the m and the B which are being calculated for us so let's go ahead and calculate our error here as well or the difference between our new y value and the old y value so our new y value we have called
1:08:24
FX and we're going to just subtract the other y value which is the grade and that will give us our error and then let's go ahead and plot this so let's do a plot let's make it a scatter plot and this is the same scatter plot that we just made where the x is study hours and the Y is grade so now we want to plot that regression line how do we do that well let's use the plot item here and we're just going to tell it to plot and what do we want to plot well we want
1:08:55
from the regression data frame what we want to do is we want to plot the study hours right but the line that we want to draw here is from our regression data frame we want that new y value that's actually what we wanted to plot and let's change the color here so we can see it a little bit better we'll just make it uh cyan all right and then let's go ahead and show this and let's see what we've got let's run this and boom there's our line of best fed and hopefully this makes sense now we
1:09:26
used that slope intercept form to calculate these new y values and we connected all those new f x y values and that makes the line and that just shows us sort of the trend line here so we can see that as study hours go up so does the grade now again we could have seen this visually without the line but that's not always the case sometimes the line is very helpful to help you identify and Visually see the relationship between these items now I told you things were going to get super
1:09:55
interesting and here's where it's all going to come together for you so you know these m and b values up here I actually want to print these out right so let's just print m and b our slope and intercept let's go ahead and run this and we get 6.31 and -17 these numbers are important because we now have all of the information that we need to actually predict what a grade will be based on the amount of hours that someone studies so so here's how to write a function to do that so we're
1:10:24
going to have a new function here let's just call it predict and then we're going to pass in the x value right the x is the number of hours that someone studied so you're going to pass in that x value we're going to calculate what their grade would probably be so our M if you remember before we had the m and the B right so our M was equal to what 6.31 it's right there above us uh 3 4 the more decimals we include the more accurate it will be the B was equal to what minus
1:10:55
17.9164 again it's these two values here and then what we're going to do is return MX plus b okay and that's going to help us predict the grade which is that value on the line right that regression line which is the line of best fit which is most likely the grade that someone would get if they studied for a certain amount of hours so let's go ahead and call that let's say our prediction prediction here is let's call our predict function and we're going to pass
1:11:25
in and say well what if they studied 14 hours what would their grade be and then let's just go ahead and look at our prediction so our prediction is that if they studied 14 hours their grade will be a 70. what happens if they study for let's say 20 hours 108 so you can see here that we kind of need to cap this because it doesn't know that the highest grade you can get is 100 but essentially studying for more than about 18 hours 95 right so if you want an A in this course you need to
1:11:58
study for 18 hours and if you've made it this far you know a lot about data science and you actually know something about machine learning as well because you just did it you used a linear regression algorithm to make a prediction based on the amount of hours that someone studies thank you so much for sticking with me through this video series check the description in this video for more resources that you can go through on your own interactively to learn more
1:12:26
about data science machine learning and artificial intelligence happy coding
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat