Skip to main content
Loading Explore guides…
Regression Analysis · Acedra
Home
Explore
Mathematics
Regression Analysis
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
639 segments
Copy
Hide video
0:00
regression analysis That's all about data It helps businesses understand the data points they have and use them specifically relationships between data points to make better decisions including anything from predicting sales to understanding inventory levels and supply and demand Understanding this we have coming for this regression analysis tutorial Now before we go ahead with this session I'd like to inform your life that we have launched a completely free platform
0:28
going Greek Learning Academy But you have access to free courses such as the iCloud Investors marketing You can check out the details in the description below Now let's are quick glance of the agenda They will start off with linear regression algorithm Then we will learn about the logistic regression algorithm So let's start off of this case study Now let's say you are conducting a survey on a bunch of college students on You'd want to understand if
0:58
there's a relationship between those CDP off the student and his or her g r E Score Now the problem statement is basically this So you were understand of the students who have ah higher CDP on certain to have a higher G I t score and similarly If students have a lower CDP then would they have a lower grt score Is the problem statement now To find out the solution for any problem you need to gather data So that is what you go ahead and do So let's say you go to around
1:25
500 odd students on ask them their jihadi score and also ask them the CDP and then you create a table out of this But if you just look at the stable would you be able to draw any sort of inferences from this Obviously no So this is where you do some sort of data Visualisation So or here You've got this jihadi score and you've got this CDPF Now what do you do Issue will map the g r E score onto the Y axis and you'll map the CDP onto the X axis and you'll build a scatter plot for these
1:55
two on As soon as you build this graph you can clearly draw inferences So what you understand us As the CDP off the student increases the G R E school off the student would also increase So they're sort off a positively new relationship between the cdp of the student and the jihadi score off the student So with this you can clearly state that if a student has high cdp then he'll have Ah hi Jerry Score Similarly if the student has a low CDP then Hill sort of have a Luigi Ari score
2:25
Now this inference was quite easy to draw from this visualization Now what if I actually change the question So let's say if I ask what will be the Grt scored off a student if his c g P s 8.32 now if you just look at this graph or here on def I see that the CD piece 8.32 would be actually a bill to tell what Toby is Johnny score Now that would be difficult And this is exactly where regression analysis comes in So let's understand what exactly it's regression analysis So as
2:55
it has stated over here regression analysis basically helps us to understand the relationship between two or more variables by creating a predictive model Now this simply means that you have a dependent variable and you have the independent variable So as we saw in the previous graph we had mapped the jihadi scored onto the Y axis and the CDP onto the X axis But when it comes to regression analysis will do something different So in regression analysis
3:20
you basically have a dependent variable on one or more independent variables Now we will map this G R E score as our dependent variable and will map the CDP as our independent variable And we will try to understand How would the Grt score very with respect to the CDP Or in other words we obviously trying to understand How would the dependent variable change with respect to the independent variable Now since we already know the value of CDP which was equal to 8.32 so I
3:48
would have to take this value off CDP plug it into the X variable and then I'll get the value off White with his out regression analysis works now the simplest category off regression analysis IHS Lena Regression where we basically try to understand the lean your relationship between the dependent variable and one or more independent variables and the important thing to note in linear regression ist your dependent variable needs to be a continuous numerical value so And
4:16
this is what we have with respect to what problem Statement So over here Jari score is our dependent variable which is a continuous numerical value And we are trying to understand how does it change with respect to the CDP off the student which would be our independent variable And when it comes to linear regression we basically have a street line So a straight line is a linear equation And that is why this algorithm is known as lean your regression So we have why and
4:42
we're trying to understand how does why very with eggs So this be what you see So this is the coefficient off X and this is basically this loop off the line And this a what You see this is the intercept off the line on the Y axis Now let's accept this concept Tofino regression to the previous problem statement So we had the jihadi school of the student map to the Y axis and we had the CDP off the student map to the X axis Now we know the CDP of the syringe was 8.32 So let's see if this value
5:12
here is 8.32 Then the corresponding value off the G R E score of the student would be around 308 all 309 But the question over here which and Isis How do you know that this is the best fit Like now let's see if I draw another line like this on the CCP of the student is again 8.32 So for this value off 8.3 to the G r E School of the student could be something girls So this time the Grt score could be around 312 or three Island 13 So as you see when it seems the equation off the
5:45
line over here the value off Why also changes Let's say I draw another line like this So again the value off excess seam But the value off by will change So this time though Grt scored would be around 305 Now similarly we could have a million Are the lines passing through the's selloff points So party know which is the best fit line right So to understand that will take this example here So there is something less measure off good fit Now instead of taking all of those points will just take the
6:15
selloff six points forward here and I'll start by drawing an arbitrary line over here now these green points which you see So these are the actual observed values for the value off X So when the value off excess equal due to the observed value off by this five But then again the value which is being predicted by this line would be around 2.2 So there difference between this observed value off my and their predictive value off by by this line is what is known
6:46
as the residual or the editor in prediction And similarly you'll have a residual for all of these points over here Let's take this point now here the X value is equal to two and the observed value off by s four But the predictive value off Why would be around 3.8 Similarly if you take this point X value is four and the observed value off by a stoop But the predictor value off by over here is five And if we take this point X value is five absorbed value off why seven and
7:17
the prick their value off wires around the 6.5 right to these dotted red lines Would you see these basically indicate the residuals Although editor in production and get that best fit line don't you agree that we would basically have to reduce these individuals as much as possible And this world we have something no other residual some off square So what we do is will go to this first residual and will square this up Then we'll go over this second residual will square this up
7:45
go to the third residual and then will square this up and we'll do this for all of the residuals And then we'll take the summer off all of the residuals Now when we take the sum this has to be as Lewis possible And whichever line would have the bluest value off residual some more squares that would be the best fit line Now let's extend this concept of the previous problem statement So this was our problem statement where the Grt School was on the Y axis and the cdp of the student was
8:12
under X axis And we've got these three lines over here So for the first line the residual sum of squares comes to be around 28 For the second line the stool for most squares comes to be around 22 And for the third line though this dual samo squares comes to be 24 Now when we compare all of these three residual sum of squares the second line would be the best fit line for our problem statement So now we've got the best fit line on We only know that or linear Line would
8:41
have an e question associated with it which is basically by equals MX plus C And that is the question or here So here why is basically the Grt School of the Student on X is the CDP off the student and these two which you see the n value and the C value this we've got from the best fit line So the M value which is the slope of the line comes to be around 6.8 and the Y intercept comes to be to 61 now to get the Grt School of the Student all we have to do with black in
9:09
the CDP off the student which we already know so will plug in 8.32 into this equation and will get a final value off 317 So this is how you were able to get the Grt score off the student by just knowing the CDP off the student And this is how linear regression algorithm books
9:28
Now that was a simple linear regression where we had one dependent variable and just one independent variable I will extend this concept into something Yunus Multiple linear regression So in multiple linear regression will have multiple independent variables instead of just one independent variable So to understand Dr Will take this day does it a war here So this is the famous empty cars D does it that we have these columns So this is the MPG column Then we've got the weight of the car the
9:57
horse part of the car the number of cylinders in the car in the number of girls in the cars And we want to understand how does the mileage off the car very with respect to all of these other independent fee edibles And since mileage is our dependent variable will denote it with y and all of the other independent variables would be denoted like this So wait I'm denoting it with x one horse par with X two number of cylinders with X three and number of girls with X four And this is
10:26
the final equation off this hyper playing over here so we'll type in Why equals x one plus b x two plus e x three plus d Export Place E And this would be the equation off the hyper plane So implemented renewed aggression algorithm will be walking with this Diamond Sutra Sit on Let's have a glance of the description of this data set so the studio said comprise off 10 columns and there are 53,940 does Or in other words there are 53,940 diamonds So we've
10:56
got the price column which basically notes the price of the diamond and US dollars Then we've got carried which denotes the weight of the diamond Then we've got God which is for the quality of the cut You've got all of these different columns over here Then we've got X y and zed which basically you know the length of the diamond the birth of the diamond and the depth of the diamond and millimeters Right now before we go ahead and start of by implementing the
11:21
linear regression algorithm let's do some data manipulation on dinar visualization and let's understand the street as it properly So while start by looking at the structure of the state as it and to look at the structure of the data set I use this str command And inside this ill passengers does it which is diamonds So I'll type and diamonds over here Right So these are all of the columns over here So you see that characters on your medical column and then we've got cut
11:46
color and quality which are factors Then again we've got on your medical columns over here right So this is the structure of this So we saw the structure of the data set Now we'll go ahead and perform some data manipulation operations So let's say I wanted on your subsidy off this entire day does it where the cut off the diamond is equal to premium on The price is greater than $5000 So for that I'll be using Go to ply package So let me load of the diplomatic Ege so I'll type in
12:14
library off the Flyer Onda We've got the filter function from the deploy package so I'll started by giving the name of the data set which is diamonds on Then I use this pipe operator Now this pipe operator is used to connect different objects and functions together So with the help of this pipe operator I am connecting this diamonds date asset with the filter function and inside the filter function I'll give in my conditions so the first condition would be the cut off The diamond
12:43
needs to be equal do premium so I'll type in premium over here and then I'll give in The second condition which is the price of the diamond needs to be greater than 5000 So I'll said the condition which is priced is greater than 5000 And I also this in our new day does it on the metas Let's see premium price Let me have a glance of this So have a glance of the data said I'll be using the view function And inside this I've given the name of the deal aside which
13:15
is premium price right So are of those 53,000 diamonds We have extracted only 4717 entries where the cart off the diamond is equal to premium and the price of the diamond is greater than $5000 Right If you look at these two columns closely then you see that in the CART column The value is only premium and in the price column it is greater than $5000 now Similarly let's do some other operations Now let's say I want the length off their diamond to be greater
13:48
than four millimeters So let's go ahead and given that command again I will start a bag during the name off the data set After that I will use the five operator and then use the filter function andare Inside this I'll get the condition so X needs to be greater than four I'll store this and diamonds on the score length Now I'll have a glance of this view off diamonds length So as you see over here if you check this length column which is X you see that the land is greater than
14:23
four millimeters so these were some basic data manipulation operations Now we'll go ahead then perform some data visualization operations using the G plot to package So I loaded 30 deployed to package I'll type in library off cheesy plot to over here Now this data said BC which I'm looking with their diamonds Yes it is also part of the DJ plot to package So if you want to implement or those same court in your system then you'd have to load the cheesy plot to package
14:51
first Now I want to understand their distribution off this price column So I understand the distribution of this price column I'll just go ahead and build up his 2 g So this is how we can build a history Graham I'll type in Jeezy Blood and I will assign though diamonds Data said onto the detail you So once the Data Lear is done I'll go to the Easter ticks clear and I will map the price column or into the X aesthetic Now I am done with the detail years so have mapped the
15:21
diamond service it onto the date all year and I've also map the price Call him onto the axis to take Now it's time to determine the geometry geometries basically the type off Lord which I want to create on dull I would create a history Graham So to create a history Graham we've got the jump hissed a gram function No I'll just hit Enter and we've got this really beautiful plot over here So let me zoom this
15:44
right So as you see there are a large cell of diamonds so they are around more than 15,000 diamonds whose price would be less than around $1000 Right And there are very very few diamonds so there would be no living around 20 or 30 or diamonds whose price is greater than $15,000 So these were some of those simple inferences which we were able to drop by building this instagram
16:09
So now that we're done with the basic DVR manipulation on data visualization will head on to the main part where will be building the linear regression algorithm Now we want to understand How does the price off the diamond very with respect to the carrot off the diamond So let's see if I given a carrot value off around one point three so the carrot values 1.3 Then I'd want to know the exact price off the diamond This is where linear regression comes in Now before
16:39
we build a linear regression algorithm we actually have to divide our data set into training on the testing said so we'll be building our model on top of the train set and will be evaluating the results on top of the test set So to split the data set into training and testing sets I would need the CIA tools package So those CIA tools packets gives me the sample dot split function So let me just type in sample dot split over here and the stakes in two para meters So the force perimeter
17:08
is basically the column with respect to which I want to split my data set on day I want to split this data set with respect to the price Call him So I also like this Price called him over here and I will give in the split issue So I'll said the split re sure to be equal to 0.75 and I will store this in a new object and I will name their object to be split tag Now this retail this commanders executed you guys will be wondering Why do you have to divide a data set into train on and
17:39
just set So let's take this example to understand it better So let's say you have ah math test tomorrow and you haven't learned anything at all But your friend luckily steals the question People from the principal's room Onda When you see the question people you see that data five questions and you perfectly learn all of those fight grecians Now there are two situations If all of those five questions would appear in the exam then you'd east the example You'll get five or a five
18:08
bite water None of the questions appear in the example You'll get zero R A fire right So this is what I'm trying to tell you So we'll have to divide the data set into training Investing set should have to train your model on one particular set and you'd have to test it on another set on If you don't do that and directly go ahead and build your model on just one particular data set then there are very good chances that it might feel on a new data set So this is the
18:36
reason why we divide into two particular data sets So here we've got this play attack Now let me use the head function and see what is there in the split attack
18:45
So split on the score Dag Now as you see you just got a bunch off through values over here So what dysfunction basically does is it assigns 75% off the records with the true values and 25% off the records with the false values Now using this true and false values will divide this entire data set in tow two parts So for this I'd have to use the subsidy function So I'll type in subsidy over here And the first Barham it would be the name of the details it and then I use
19:14
this plate tag I'll just type and split tag over here And wherever the value of split tag us equal do true I will store this in a new data set and name that data said to be equal to dream Similarly I'll use those subset function again and I'll pass in the first parameter which is the name of the data for him On this time I will said those split back to be equal to falls So wherever the split tag is equal to false I'll take all of those records and I'll store the window That said Now that God or
19:47
Green said And the That's it really Now let me have a glance of the number of fruits in the train set and the test it So I'll type in Andrew Off Train and Andrew off best So we see that there are 42,126 entries in the train set and there are 11,814 in prison though that said so now it's time to go ahead and build the morel on top of the train set and a bill the linear regression model We've got this alum function so I'll just type in Elem And the first parameter is the formula where we given the
20:20
dependent variable and the independent variable So my dependent variable s price After that I'll given though they'll assemble and then I'll given the dependent variable which is carried So I want to understand How does price very with respect to the carrot So whatever value is given on the left side of the tell a symbol that will be taken as a dependent variable and whatever values are given on the right or the tail assemble all of them would be taken as the independent variables This
20:47
is my first perimeter which is the formula And then I'd have to give in their data set on which I'd have to build a model So only tell you we are supposed to build this mortal on top of the tree and sit And I will store this in a new object and I'll name it us Let's see Linear mortal one Let me just print it out over here Lean your model one Right So this is what we have over here So call You see that we have the formula price with respect to carrot and it is built on top
21:19
of the tree inside and beneath that we've got the Y intercept and the M value for this independent variable so x discarded and the n value or the coefficient off this car It s 7894 and the white Indus it So why intercept for the question A sick will do be X would be the e value we're here So we've got the y intercept and we've also got the M value Or here Now that we've built the modern let's go ahead and predict the values on top of the test said So I'll type in predict Now this takes in two
21:53
perimeters so the force perimeter would be the model which you've built and the second perimeter would be the data set on which were supposed to break the values And I want to predict the values on top of the that's it And I will store this in Let's say my results So we've got the actual values and we've also got the director values Now I'll go ahead and combine these two So to combine these two I'll use the sea by and function so the actual values are present in the price column off
22:22
this test data set So I just named the Scala matters actually and then I will create another column which is predicted on this Richter Values are coming from my result and I will store this and final day Doc
22:41
Now I'll have a glance of this So view our final leader So we've got the actual values off the diamond and we've got the predicted values of the diamond Great So let's save This is actual value 3 51 depicted us 54 So there's a huge editor in this prediction over here Now we'll go ahead and find out the residual Audie error in prediction so that would be basically actual minus predicted That's how we find the editor in prediction No uh have to convert this into a data frame first So this is
23:13
actually a metrics So I use as dot data for him and I'll pass final data into this and I will store this back to final later on Once we are done with this I will go ahead and subtract the pretty values from the actual values So final data dollars actually minus final data dollar predicted So I'll cut this out I'll be STIs inside these round braces and I'll store this in a new object now name the object to be equal to zero and I'll bind this era back to the same day
23:50
Does it So see bind off final data and I am binding it with error and I am storing it back to the seemliness said witches Final data It's a God the actual values They've got the predicted values off the price off the diamond And then we've got the error in production So these are the individual arrested deals with respect toe each point Now we want to find out the residual sum of squares So we have something whereas the root mean square error which basically
24:21
gives us the average error for the entire model So as it is stated in the name root mean square error So first we have to find out this crowd off all of these errors so I'll type in final Data Dollar editor Let me got this out and I will go ahead and find out those square off this No once this is done So I have got the square off all of these editors Then I'd have to find out the mean off it great And finally I will go ahead and also calculate the square root off it I'll pieced it
24:59
inside this and I use the S Q rt function Right So root mean square off error This is what I'm finding out and I gotta root mean square value off 1002 121 which is actually quite huge So this tells is that the model which we build is not actually accurate for us to give us the right results and I'll store this and let's say R m a C one So this was a simple linear model where prices are dependent variable and carrot was our independent variable Now we'll go ahead and build
25:35
a multiple linear regression model so this time will use multiple column does our independent variables So let me go to this data set and let me use the Ellen function again
25:46
So price again would be my dependent variable And my independent variables would be let's say X y and Zed which are basically the length width and the depth of the diamond So I'll type in X place Why bless said Andi I am building this morning looking on top of the trains it so data would be trained and I will store the Seine linear model toe I have built my model and I go ahead and click the value so I use the credit function on the first barometer would be the model which
26:19
I have built which is basically linear too And then I will given though data on which I'd have to break the values which is new data So I want to protect the values on top of the best said and I will store this and my result to I have also predicted the values Now it's time to bind again the actual values and the predicted values So the actual values are present and test dollar price and the predicted values are present and the my result do Right now let me give them their
26:54
proper names So I'll given actual over here and I'll also given predicted over here and I will store this final data to So we have our new final date already Serve You are finally got to It's so for the second model which you've built where we had three independent valuables These are the actual values of the diamonds and these other predicted values of the diamonds Now let's go ahead and find out the error in prediction So before that we'd have to again convert this
27:29
into a data for him So as dot data stream off final data $2 This is has to be final data to and Duyvil store it back to final data to all right now it's time to find error So final data to dollar actual minus final data to dollar predicted I got this out and I will store the send Eddo Duke Now it's time to bind the era back to the original deal assets So see bind off final data to and I am finding it with two which this store again and final data to So we've got the actual values
28:16
though predicted values and the other in prediction Now it's time to find out the root mean square aero for the second model So final data $2 at 02 I got this out We sit inside this let me take the square of the Arab one side B C square off the head I'd have to take the mean off it And once the mean is also done I'd have to take the square root off it so I'll type in sq rt over here Right So this time though fruit means Guerrero comes to be around 1574 Andi I will store this and
28:54
let's say R M s E to now let me print out boot our embassy One andare embassy do over here right So for the first model the root mean square terrorists one due to one And for the second model the root mean square Harris 1574 So for the second So for the second model the root mean square error is higher This means that the first model is considerably better than the second model So this is how we can build my tipple models and compared the accuracy with each other So we have
29:26
successfully implemented the linear regression algorithm in our language Now it's time to implement the semen bison and Facebook Bush will be working with The ideas do does it So my first us would be to load the bond us luxury so I'll type in import pond us a Speedy Now I'll be using the read underscore CSB function so I'll type in pd dot reid underscore CS Week and inside this I'll give in the name of the file which is basically IRAs dot CS week and I will store this in a new object
29:58
Now I landed Object us Iris No I'll have a glance of the head So Iris Dart head And this week amid off first Firoz which are present in the city does it now This day does It is about their different species off the iris floor which are Sentosa Warsi color on Dwyer Janica and these are the different features off these species So I've got the SEPA length CEPAL worth Battleland's and petal word Now for this model I'd want to understand How does CEPAL length off the iris flood Very with respect with
30:30
Opie talent off the iris floor so separate Lent would be my dependent Variable and petulance would be my independent available so I'll go ahead and divide this entire data set into their dependent variable and independent variable So I will store the separate length into White Do that I haven't just given the name of the data for him which is Iris and I'll give in double braces over here and I'll put in courts and I'll give in the name off the column which is separate
30:58
lint So let me just type and CEPAL underscore length over here So I am extracting this separate length column and I am storing it into the variable white Similarly I will extract this Battleland's column from the Irish data set and I'll store it into X given to braces over here And I will give in the name of the column that just battle underscore length and I am storing this into X So I have my dependent variable and independent variable ready Now it's time to
31:30
divide thes two into their training set and the testing said so for that purpose I'd have to import the train Test split mattered from the SK loan package So from SK Loan I thought it would be model selection so I'll type in Model underscores election import off train best split so I'll go ahead and import this matter from the SK loan package Now what has done I will use this function so a train tests plate I will pass in the X value and the Y value and then I given the test
32:07
size so the test size would be equal to 0.3 So this means that 70% of the record score into the A train set and the 20% of the records going to the that's it And this would basically give us four results So let me just type of those results so the first result would be extreme and then we've got the X test After that they've got white green and then finally they've got white dust So what are these four variables so extreme As basically the training set off the independent feasible than
32:40
X test is the best set off the independent variable Then we've got white rain which is the train set off the dependent variable And then we have the whiteness which is the best set of their dependent variable So these are the full results which are given by train test split So now that we have all of these it's time to build early new regression model on top of the train set So I will import the linear regression model from Escalon So I'll type in from SK loan dot lean your model I'd
33:09
have to import linear regression So now that I import linear regression I have to create an instance off it and I'll name that incense to be our right So I have successfully created an instance off this linear regression model and I'll go ahead than fed this on top of the tree and said So L idot fit And I am fitting this on top off extreme And why train Right So we have for this model Now it's time to break the values on top of the test set so they've caught the credit
33:47
function So allow dot predict and will be predicting on top of the tests It and I will store this end Let's say why underscore Fred Now we've got the actual values in the predictive value so the actual values are present in White test I'll type in white us dot head on Then I've got depicted values which are presented wipe red but his own number I worry So I would have a glance of the first fire records that go from 0 to 5 actually Right So these are the actual values off the
34:22
Zeppelin and these other predictor values off the separate lent to the actual value is 5.7
34:28
The value predicted by this model was 5.9
34:31
The actual value of 7.4 picked it was 6.1 and they've got 5.1 which was picked last 4.9 Right and free Take this value Seven It was baited US 6.15 so I've got the actual values and afflicted values Now it's time to find out the editor in prediction So I load the mean square error from the metrics office Kaylan So I'll type in from SK loan dot matrix I would basically be importing the mean square So now that I import it I will go ahead and find out the means quite other of
35:05
this model And inside this I'll pass in the actual values which are present and white pissed And then the second barometer would be low predicted values which are presented wipe red So that means quite letter comes out to be 0.19 which is actually quite less so It seems that this morning with you build us quite good So in linear regression it's one of the most popular algorithms and production Because it can be used for classification it can be used for regression It also happens
35:39
to be one of the core algorithms that is used by many of the algorithms Logistic regression is a linear regression based model right If it takes support Vector machine another very famous algorithm and data science It's a linear model Or if you take uh if you build do linear models using been the data what it called been B I and binned data sets then the modern that you get behaves like a decision tree right So many of the other and Gotham City are very
36:14
popular in the industry Internally there linked to linear mode it's you can spend your entire life's time studying linear models It has so much of depth and breadth that it takes a lot of time to actually get used to all the six So today I'm going to start by introducing you to simply need models the core concepts and then subsequently and leave some advanced discussions for the next visit where we'll be covering F empty feature Injuring model tuning there will discuss some further at
36:53
once concepts only near Mali's only So I hope these people have loaded by My team has loaded the data set everything I'm going to give you three chords today offiicial covered one in the class The other two is for you to go and explore further Okay
37:15
Based on what To discuss Nicolas This property is accessible to all of you Can you access this property I think she's going toe You can You can Okay The court is also available The people the The court of the data said you should have received three chords All right let's start as the name suggests this algorithm is called linear regression on the reason why it's called linear regression has names of justice It's based on the concept off a line a line a plane Ah
37:55
hyper plane We call it by these names
37:58
So to begin with I need toe ask you whether you know how a Linus represented
38:05
line line in mathematics Are you uncomfortable with this All right so let's start Imagine I give you two dimensions This is X excess here One off your independent attributes We call it columns in our day Different right So loaded a data from our table into my data stream on one of the Collins in the data frame I call it X This is my independent variable The other column which I want to predict the target column I call it way It could be anything Why it could be the mpg on it X
38:49
could be the weight of the car The more the weight of the car the list of mpg right So why is my dependent variable excess My independent Imagine this X has
39:05
456 These are the units 12345 Think these are the measurements off X and let's assume
39:16
for takeoff discussion Let's assume that whenever excess one why is also one So whenever excess one wise also one That is what you're seeing in the data You take another record there you see X physical to then vice is old Vehicle is also equal to two
39:40
eight the third one Excess three Why is this also three So when I tried to draw a line a relationship between them you see it's a straight line The straight line tells you the relationship between X and why How are these two related to each other
40:03
We in data science call these things as models Models are nothing Berck Sur faces in your feature space What is a feature space feature Space This collection off your independent attributes under different attributes You're trying to explore how this why and this ex interact with each of that interaction is what be represented as models Some mortals are nothing but lines surfaces
40:35
Hyper sir Faces in your feature space 70 built k nearest neighbors Okay In Kenya's neighbors you have different classes of data points Suppose these are different classes Okay on Okay let me make it slightly more conflicts What can nearest neighbor does this we call it breaking a mathematical space into regionals It breaks a mathematical space into regions So it says this region belongs to the circular point This region belongs to the strangler point and yes things
41:14
on the wrong side Getting nearest neighbors splits your mathematical space into regions Those regions are called Warner regions So this imaginary surface is the modern in Kenya Same linear regression This line which represent the relationship in x and y This line is my morning So I want to predict the value of why given the value of X what the line is saying is why is equal to X So the prediction ISS whenever excess some value Why will also be the same Can I write
42:01
this expression s Weichel Dwan ICS Does it make any difference One into its this one
42:12
In mathematics it has a meaning This one is nothing Bird This angle that line makes with the X axis the time off that angle off that angle into X whenever Why is equal to X This angle will always before the five degrees Okay and in trigonometry
42:44
ban off 45 years old This one I met him wondering What is this technology Time Petain or literally Jardins actually told comment Since
42:56
look at this when we started in school became across trigonometry in trigonometry became ultra signed Teeter costed out And Rita What are the scientist accosted Attended one of these things These are names given to certain ratios So tank Tita is time means the ratio between we sure does d Viber dx How much white changes whenever exchanges by one unit that issues court Hamas Why changes when exchanges But one unit How much white changes in this stuff when exchanges by 100 so one by one
43:41
is equal to one de Viber dx How much white changes by small change in x one Entertainment eggs that ratio is called tan Tito sounds very high five but that's what it is Okay So venom er both of them synchronize whenever exchanges by one white in Just buy one It will always happen when this angle is 45 So we say that China fortifies one So when you say 10 afford every developer decks will be one by one Little visit garlic Let me modify this a bit So right now this expression is
44:25
represented as am Okay This is a standard practice very represented as m So why equal toe a mix This the expression mathematical expression off this land Suppose I put a line here okay And it's parallel to this line
44:49
It's battling toe this line Okay which means
44:52
this line is also making 45 degrees toe the line Parliament X axis I'm not going to show this link It okay just to make it less cumbersome So I have another line here This is L one This is L two No Can I see way critics
45:17
then x zero Why is not zero wise to Glenn exist One Why is not one Why is three when access to why is not too wise Four So we add a constant toe This this constant is equal to two right now for this life This constant is nothing But what is the value off wife and x zero What is the value of y given x zero That s Corsi
46:00
It also goes by the name off Intercept
46:04
Where India Mathematical space on the white dimension
46:08
They're on the by dimension Does the line intercept or cut the Y axis So this is the general equation Off line Weichel toe emits Lissy It tells you how why and excel related If you look at this line L one If you look at l one
46:38
then this becomes zero So why equal time makes if you look at l two l toe This becomes not zero but to so this relation continues to remain missing In both the cases the M is seem em is nothing but turn off the 45 degrees we call it slope Right So this is the gender Rick equation one way off representing a line y equals MX plus c So in this expression why is given in your d desert It's a target column X is given in your data set it is the independent columns they live in and attributes from this
47:37
data that they have given to the algorithm
47:40
The algorithm miss found out for you the am and the sea We call these coefficients
47:48
coefficients off immortal this M and the sea
47:54
They reflect the relationship between y in X seniority does it so the M and the sea is what form Stay immortal for you
48:10
Shall I complicate A bit more will complicate a bit more Suppose instead off having one
48:25
Or let me let me just continue with this only so you have one target variable boy one x In this case 10 off 45 happened to be one But suppose you're Linus like this It's a straight line but it's not At 45 degrees it's a 15 degrees Right then in that case also the question off the line will be Weichel Tow em x The reason why I'm not putting CC zero here C zero you can put see what's easy And this angle m this value of him will not be one It will be 11 live in its 44 degrees
49:13
It's less than 44 degrees So tan off 15 degrees into one into X plus c This will be a question of this life
49:28
What is Stan Data tended does nothing but how much white changes for one unit change in X we call it then Okay All right let's complicate this a bit more Suppose in stuff only one variable x No you have X one and you have another variable x two and you have only one dependent variable way In such case the algorithm will find out the relationship between x one and why X to end way So it'll express that relationship as why equal toe m one x one plus m two x two And
50:17
there will be only one constant Because now
50:20
in stuff having a line you will have a plane near the plane go through the Y axis At some point it cuts the Y axis at some point So now you have to adjust one edges on X to the other Edge is on X one The two edges may have different slopes Different angles with X axis Those are reflected over here is M one and the C continues to be the same scene Where is the plane cutting the Y axis Which means what is the value of why when X one and x two or zero That is called interested
51:08
Good one You have zero Also will be called a hyper plane We don't know how it looks but will be 11 Yeah So human minds can imagine when the three dimensions
51:27
but supposed to go beyond three dimensions There also the linear models work Those planes are called hyper planes Hyper planes How do they look We can't imagine So we I don't know that But they will also be one single plane and there'll be no ups and downs No curves Nothing will be straight plane What do you mean by straight plane in four dimensions Lawyer It led the same properties as the properties of a plane in three dimensions These excellent actually in the same can
52:03
be in a different right 90 degrees this angle in linear models this is a three dimensional isometric view This angle is always 90 degrees We say that there are struggling to each other Look at this laptop Okay In this laptop now this is my ex one dimension This is next to dimension There is a plane This plane is your model This plane is making some angle I can't make it right now I can Right It will make some angle with excellent access Will make some analytics to access This is named to
53:00
this is my M one So at what angle The plane is two extra Next one will design the value of y Okay In this case the value of seyval B zero Because the plane is starting the Y axis y axis is perpendicular like this It's touching the Y axis at the origin The zero point some Imagine I can remove this If this also you gotta have laid I could remove this OK and imagine I'm putting it in some angle Cutting the Y axis at some point Then this angle this angle is my M one m two the point at
53:39
which the plane cuts the Y axis Must see We cannot imagine beyond three dimensions how the planes will look But they will have the same mathematical properties as a plane entry range a point that you should probably keep in mind It will become useful to you down the line when you're doing be planning since a foot the number of dimensions off model This is the model right now The set of face the number of dimensions How many dimensions is this mortal Two dimensions lent
54:15
in the breath The model is merely to sanctions the feature spaces three damages So the number of dimensions in the model will always be one less than the number of dimensions in the future space Right All right Um
54:35
you can call it anything like that X y z Okay that's what I say You want to call it send your colleagues It doesn't matter Just replaces Basically these are independent variables This is a target So we're trying to understand how the target variable is influenced by these I see you have feed off a cliff So how is the model dimension
55:08
No X one extracts 33 independent variables So they've on target variables so that makes it for four dimensions Martin will be three dimension in darkness on however it looks I don't know what will be left 33 in just
55:31
we don't know how the hell did you So that's why we don't talk about it but the properties of that model will be similar to properties of a plane Alliance on
55:48
That is what the algorithms resume All the algorithms as you that you're independent variables Okay Supposed to have a data free in the data frame were multiple columns These are independent variable C I want every to every three and this is my target All algorithms assume that independent variables are independent off each other
56:08
They don't influence easily The only influence the target that's what they resume But in real world that is not true Suppose I want a predict mpg off a car given the weight of the car Given the horse port of the car we know that they're related
56:26
so we can build a model But we also know that horsepower and weight are also related to each other more the weight of the car more The hospital They're positively really But the algorithm assumes that they're not related this problem in linear models it's called the problem off Colin Garrity Their assumption is each dimension is independent off the other but in reality they influence each other That problem is called Colin Charity Okay this calling Garrity we're
57:01
independent dimensions interact with each other They're not independent This Colin charity can lead you to a problem Venue production Isar models The models may be less effective Less vertical predictive power
57:18
then required Toby because of the culinary So one of the things that we do when building models ISS we see what the Colin charities between the dimensions If the core linearity is very strong they're very strongly related to each other The independent variables Then we may have to resort to some techniques I suppose
57:42
this is my horsepower on this is the weight of the car on def A plot
57:48
A graph between these two Can you tell me how they look This is weight of the car This is hard sport of the car If applauded graph between these two Take all the data points You'll see some linear trend Okay All if you know what is our value Condition of Correlation R Value started Six You don't start a six Uh are really measurement off how these two variables related to each other How strongly you will see that our value comes very close to boss to one in this case the maximum it
58:32
can be is one suddenly very close to postural Okay which means these two levels are not truly independent off each other but the algorithm wants all the dimensions to be independent of feasible So in such cases we resort to a technique called dimensionality reduction which is not simply dropping the dimension I may convert them in the composite dimension both the dimensions put together I create a synthetic dimension and I use that synthetic dimension in my model
59:09
building rather than these two This is the horse part of the way is the same almost like it's in visiting If that removal is an option to you when the relationship absolutely line but you will see that it's generally not a line it's a spread on The spread is also covered So the trenches I drop should I drop this or this That depends on which dimension is more prone to It arrives when you calculate given me Katherine the later So it's of being all that analysis I'm a convert into a synthetic
1:00:01
dimension using a technique called PC a principal component Analysts No no that's what I'm saying That is our brain Our brain can see military dimensions But the algorithm can one in working 300 damaged All the 300 dimensions mathematically will be orthogonal to each other Okay you please keep this caution part for a minute I want to cover some ground before I answer that question because that's a very important question you're asked All right let's move on
1:00:51
Uh the term regression indicates that we're going to predict some values Whenever we build models to predict values we use the word regression
1:01:03
the term linear No you understand where it comes from so you're linear Model is nothing but a combination additional off waited variables This waited variable this another waited variable You're adding the two to protect the value fly It's a linear combination Off one or more variables with weights The weights are called coefficients Okay so once again at it beat What is this M one m bonus How much white in just whenever expunging just by 100 Keeping X to constant What is his m to
1:01:41
M to us How much white changes whenever extra changes were 100 keeping X one constant
1:01:50
Okay So this is the relationship between x one and why this is the relationship within extra when boy the relationship is how much whiting this whenever input changes But one of it that is a relationship and that relationship is what we call linear models We're already seeing this Brenda more So before we start building our models before we start building our models we actually need to analyze the data Sick on what I mean But this is suppose this is my data Sit I'll go
1:02:31
in analyze Between this I've even on target Ivy to and target Ivy Tree and target ricin off These is strongly related with the target If supposed ivy too On the target
1:02:47
the distribution is scattered all over the place
1:02:53
It's like a cloud The data points of spread like a cloud There's no trend you're seeing in the spread then in that case I ve to It's not a good predictor off the target How can you say that Take any value off ivy too Broad Street Ling For that value Five me to your multiple values off target Which one should I take I don't know which one to take
1:03:24
So we say that I've even ivy to in Ivy to sorry we say that I ve to independent variable too It's not a good predictor off the target On the other hand if suppose between say I V one
1:03:43
on the target we see the data data points scattered in some form for trend Then we say that I've even is a good predictor off the target So the first filter that you'll apply in your data set is off all the columns given to you which calls strongly relate to the target Those columns which don't relate strongly with the target you're going to drop them from your analysis You're not going to build your linear model Okay
1:04:18
So the relationship could be posted or the relationship could be negative It could be downwards Also it I want a strong relationship Doesn't matter It's this feel that way I don't give but I don't want to include any columns in my model which have this kind of a spread Worse is the way Okay You I did Linear models have built one leaving you see linear relationships If you see nonlinear relationships then you might want to resort to nonlinear models Okay Lenient Morton's They expect
1:05:02
linear relationship between Why on the independent variables
1:05:08
then your model's predictive power will be very hard However as you'll see now those are all very perfect conditions in the real world We hardly have such perfect conditions right So sometimes even when the relationship is not linear we go up building the normal we'll see them Okay how do you measure the strength of the relationship between independent variable and the target For that We use a metric called coefficient off Correlation How these two variables the
1:05:41
target on the independent variable how they call relate how they relate to each other which we express using a character called our I'll tell you what this art is this our metric can have a value of plus one
1:06:01
minus 12 plus one In this range the least it can have its minus one The backs it can have a values plus one
1:06:12
It cannot be beyond the strange Our menu can be Zito Canvas Ito We now spread like this Your R value will be very close to zero When you're a spread like this you are really will be very close to minus one when you have a spread in the other way you're are very very close to plus one Okay so I will use an indicator off how strong the relationships are So let's understand this article You're slightly more detail the formula for our values This
1:07:00
this is a formula for our value Okay
1:07:03
so shall we begin Don't worry about the formula Don't wherever the formula understand the concept all the formulas are there in the library We just need to call those functions and getting things done But get the concept in please Yeah Isn't it our values A measurement based on the spread off the needle No
1:07:37
Have you heard of scaling Scaling linear model is one more linear regression where scaling has no impact Whether it's killed it are not
1:07:49
It's going to give the same result What if we know the formula for variance What is the form lovelies It is Is I minus x bar You can call it mu the whole square And okay I use It'll do one toe and let's keep it end But as always your views and minus one
1:08:27
If sample sizes less using minus one Okay all if you okay This don't Whatever distance What is this variance That's what you need to understand that the former Thank you And I spent What is it telling you always A really good We're What is Central Valley What is X bottle What is new What is the central value Don't give the formula Don't give in the form line of the formula Central value is a value which represents all the data points that you have Okay so if I take all your heights added
1:09:15
up divided by a number of students in the class that'll be average height off a student in this class Now How good this averages First of all why did we take averages Why do we need mean and median value We do all those things That many
1:09:31
that is started Sickle answer Yes you're right But many times in data science will come across a situation where you have a function and the function takes us input Only one value to predict something Now which one of the values should I put their 30 students in the class Each one has a different height right I want city order some kind off court for all the students in the class A great learning stamped court Know what size should I tell the manufacturer I can't
1:10:00
give different sizes and one unit off every size It's hard to give some size ritual represent all a few So that all if you conferred in comfortably right that is idea Okay Inside situations we need to be in for 21 value So we What we do is we find the export or the mute
1:10:20
But the question is holder level Is this expert Suppose average height of the student in this class comes to 5.5 Some of you might be six photos You might get off under doing that Boss I'm 6 ft broken I'm fighting for a 5.5 So how reliable your central values is based on what you're going to do for their actions how reliable your central values is That reliability is given by the mission of variants If the variance is too large the central values not reliable
1:10:56
Right So the variance gives you the reliability of the central values out Aladdin than single
1:11:05
So the formal off of agents Is this
1:11:08
all right Just look at the new miniter
1:11:11
This Look at the new minutes This is the case when you have only one variable x No I have two variables X and Y I want to see how they very together How they influence is they're very together That is called providence which is given by X I minus X bark We don't square it into why I minus Viber So these final take seven is export multiplied for violence labor That tells me how the data points in the feature space They very together that
1:11:47
is called Koreans Independent variables Independent variables Independent variable should not have confidence Target variable and independent variable should have strong co variance Otherwise you don't use those independent variables when you're building linear models the target variable The why on the independent Variables IV's should have very strong co variance But within the independent variables the co variance should be zero That is an ideal situation Practically never happens All right
1:12:30
No Give it the school villians No look at the formula for our okay on the top in the new minute of your cinco evidence right And if you look at it carefully the units on the top and the units on the denominator aren't seem this is also examine its export This will take Simon's This is why I am in its way But this also volume in its way But it this quiet number there's an under route
1:12:57
This is also square Tim Burton and group
1:13:00
So both numerator and denominator are going toe have same units of measurement agreed So if I have a kilogram by kilogram case than the units cancel each other out it becomes a unit lis quantity It is a ratio Our value is the issue Okay Now when you have a diagram when you have a plot like this between two variables could be indifferent Variable stargate and independent could be anything
1:13:33
When have this kind of float you will see that our values close to zero We'll see why this happens when have this kind of applaud various here trend But as X increases y decreases they're going in different directions
1:13:49
X increases when decreases that are opposing each other Then we love our will You close to minus one whenever they go hand in hand When X increases y also increases they go in the same direction together Then our will be close to plus one on It's not very difficult Understand why Let's look at this particular case closely I'm going to look at this particular place to look closely Yeah this is my mathematical space or features place
1:14:26
Both X and Y have some expert in y bar the average right expert away But keep in mind all linear models that will build linear regression models They will always go through the point where expert in the white but meat So the leaner and more than that we saw a few minutes ago this model will always go through wherever they export in the Viber meat it will be hints hints on To that point that point will act as a fulcrum All right this one particular point that Lena regression models will always
1:15:10
go through X bar Viber I'm really surprised and I find it very difficult to understand why this is not told Now look at the look at the points over here Okay let's look at these points carefully take any point Let's take this point for this point
1:15:27
What ISS excitement is Expert excitement is exploited this distance This is my expert The vertical line Excitement is expert Is this distant agree
1:15:40
all If you agree one is why am minus y But for this point it is a distance between this point and the Y bar I agree Excitement is export is postive Why am I honest y bodice posters of any Multiply these two It's like finding the area off a rectangle lend into breads So excitement is exporting The Y minus Beiber for this point is nothing but the area it covers Over here This is the area This area is a Boston radio All of you Okay let's take another point at the bottom
1:16:16
of the screen Said this one For this point excitement is export Export is greater than excite so excitement is X bodice negative So for this point excitement is eggs bodies negative
1:16:33
Why am and his wife but is also negative Negative into negative becomes pasta so all the areas over here will be posted All the areas over here also will be possible
1:16:48
I agree Take this point Excitement is exporters negative because excite is less than expert Make it two Why am it is white But it's Boston It's a positive into a negative Ligurian negative All these data points will have negative areas All these data points will have negative Indians When a total this up sick by you See sigma over here Signem in some appalled Lydia's minutes some of all the areas here all the posters on the negatives will cancel each other up The reason why
1:17:26
they will cancel you to each other Awarded each board different has same density of these points The points are distributed like a symmetrical circle So whenever our values close to zero that is because your new military is becoming zero New military becoming zero means of distributions like this for a given value of X I have multiple values off Why which Why should I take I don't know So we say that X is not a good predictor off way On the other hand if you
1:18:00
take another this plot if it becomes tighter
1:18:04
then postive areas is greater than the negative areas So I really will be close to plus one will become 11 leaving on the data points lie on a straight line which will never happen If this plot was this way all the negative areas will overwhelm the positive areas that are valuable close to minus one Right all of you Okay so this is a concept off our value So what we doing Leaning Model says we test the r values off every independent dimension with the target variable We sell it when Ledo's who's
1:18:44
our value is either close to minus or not plus one Any dimension with our of eight of close to zero That dimension is a useless change How close to zero is useless That depends on your particular problem Statement
1:19:02
Incidentally any art of alu greater than 7.7
1:19:06
minus or plus it's considered as a good variable Any variable with our value between 0.5
1:19:15
on 0.7 it requires for the research whether this relationship Israel Or is it a fluke by chance that you're seeing in your day Does it any posters here A long one
1:19:41
It could be a sampling of the way you selected examples from the population that leads to the you know the central discussion Let's just isn't so for our value to be zero Okay if this is my ex and this is not why this is a expert this is why I've are okay Expert Vibert I'm not showing you the court in it for our very Toby zero The data point Suppose I give you 10 data points The data points have to be absolutely symmetric symmetrically distributed for our failure to be zero absolute
1:20:15
symmetrical distribution in all the quarters What is it Probably that when you're doing random sampling you'll get data points in such a way that they get symmetrically distributed on this What is it Probably Broner Luisito That particular configuration of data points that we want for the universe but symmetrically distributes itself around these two experts and my bars to give you the arrival of zero that probably zero You let it happen So even in the real world
1:20:48
if there is no relationship in an expert in my bar because of your sampling sampling brings in some asymmetry in the distribution There'll be some asymmetry in the desperation You'll end up with an R value greater than zero
1:21:02
This arguably recall it started sickle fluke So when our villages very close to zeros it could be by chance sampling problem It may not be really relationship so you have to go in for their investigate whether the relationship is a medium It might leave you art Is it feet It's possible that will happen when you're sampling is completely gone Case you're done some mistake and simply when then it can happen Alright Alright Shell No one I think Yeah both cases We have to use
1:22:05
this We have to use for both the situations just like the independent dimensions and within the independent dimensions knock or those which are very strongly So it is obvious is the most only This is what you should be more street John Okay No this algorithm that we're going to use what this algorithm will do is given the data set you have
1:22:42
Okay so those this is the distribution x and y In your data set you have only one independent call on the one target and you see notice the distribution is like this So this is a good predictor off this Why But let me understand In precise terms What is the relationship in Y in X I see that there is a relationship with what exactly is that relationship precisely I want that that precise relationship between Y and X will be Arctic mortem represented by online in this case Okay The l overcome want
1:23:15
this algorithm all doors It can find out multiple lines here Multiple lines can be is shown here Right Want algorithm Will do is we'll start with some random m and random See why you could time explicit Why I call them explicit is the expression You guys I'm talking to you Not the Why is it that I am explicit E So by changing em and save whenever I change em and see the relationship between X and Y changes which means the line changes the line represents a relationship So whenever I change MNC
1:23:52
the line will change Okay This algorithm will start from sun random M and random See
1:24:02
on it will evaluate the different possible lines
1:24:06
before it finds what is called the best fit line Okay Now what is this best fit line amongst all the lines possible that it has checked best fit line is that
1:24:20
which goes through the maximum number of data points it can go through all of them
1:24:27
It goes to a maximum number of data points and at the same time it minimizes the distance between other points on the line
1:24:37
The distance between other points on this line this distance Do you see the distance over here This distance is called what it means ISS Forgiven Value of X Your line says the value of why should be this But your data available in your data set is saying the values This the blue line Here is the predicted values for the different X
1:25:05
Actual values are the black dots So anyway have black told about the line or below the line That means there is a difference in prediction and actually the difference is called
1:25:20
Okay don't mistake the word Enter Enter doesn't mean anything negative but it's used for something that you already know I'll tell you wanted us so amongst all the different possible lines Little bit only one line which minimizes the some off all these errors If I sum up all these errors across all the data points the minimum some air of it will be given to you by only one line
1:25:47
That line is called the best fit line
1:25:51
The algorithm will find out for you from infinite number of possible peace The best fit line for you It's like looking for a needle in the haystack And to do this it makes use off process which is called the greedy int distant All algorithms use under the hood a learning process a process which the makings off to find the best model for you in the given data That process in this case is called great innocent I
1:26:51
don't want to that that constant classification This is a big mission Our the same concept will come into play here also in a different way I'll tell you what the previous Yeah So we'll see that we So again once again another particle machine We're too far away from those points right Show him along Any questions Yeah So the number of points is buried Their differences more than than a country This incident it'll take action on how countries got thrown out for the distance Most
1:27:29
parity because there is this Uh yeah it only release vendor points it on the language
1:27:36
So it has to be taken together both of my equal priority But this one we one line which is satisfy the condition the driving condition ist there should be minimized That is one lee condition Okay so there are no two step process if it's only one step So let me explain this to you
1:27:56
If I Plaut okay The question of Linus Weichel do em explicit See I'm going to take one simple case MX plus e there in your original mathematical space Now ex you have way One independent variable One different Incredible
1:28:13
Okay so now we have two parameters that we can modify And whenever I modify m and C I'll get different lines This is M one and see one This is M 20 to this entry and see three earlier different lines Each line has its own MNC
1:28:34
but not all lines of good lines So if I have this line for example this is in one If I take the l one l one is made off M one see one I see that many of the data points are away from everyone on board the sites So the error even is very high If I take say the next one l two I like all this Anto I call this l toe I see that l two has its own slope its own Call this in quote blessing intercept And it also does a lot of errors but not as much as even you do I tried various
1:29:21
lines Then I come to L B If best fit Best freak line is this which has its own a m best frit see best fit on e best fit on off all these mirrors This will be the least
1:29:41
e best fit will be the least Now the question is by changing my m and C but just a small minute amount I'll get a differently So there are infinite lines possible which you will have to evaluate Order the infinite lines here to find the best fit Brute force will not work here You can't use brute force so you don't know
1:30:06
See I've actually time or anything All this
1:30:18
gives its informant across the history I'm coming to that Okay So in stuff looking at this mathematical space I can actually look at another matter which is related to this That mathematical space is between em and see on this dimension ist where better means some off all the areas put together Okay In the linear regression this terrorist calculated us Why hat that is predictive value of oy minus white actuals okay Raised to power to divided but in in just a number of data points
1:31:10
I used to want to end We call it square There some off squarely errors Some Miss Sigma squaring Data's so the better done by mortal is nothing but a some off squared errors And this number of data points you have Isn't this wonderful variants So it s nothing or Davidian's how data points very scattered across the best fit line The less of the variance more reliable the center valueless
1:31:47
The less of the variance off the points across the Morton the better the moralists Same concept comes to any differently So some of school this is nothing but variance Variants off the data points across the morning So I want to find that mortem where the variance of the data points across The model is the least you got feelings even music Cool
1:32:14
And you say they have more radios for dependent forces or is hard insulation degree of workers Okay that's wrong It's a big deal for goodness can be radius from its neighbors or change your ways Okay that she's thinking
1:33:00
someone Everything you have is thinking which is tomorrow So if you're blocking the way I want I'll take your question on Just hold on a minute I'll take the question here
1:33:34
Written 1.90 extender division sent in this side The data points might still have noise in them which causes spread across the model you're talking about old class which is lying 1.9 600 Division greater So by removing that yes you're cleaning the data But between the 1.9 600 division still you have noise That noise will appears scattered around the mornings in LaPierre's variants around the model way Anybody with Actually no You have asked a very very important
1:34:21
question which I will just answer this because this is a very important question which will be helpful down the line Just hold on a minute I landed first Okay Now if I take any access any variable X it has a spread Okay this bread variance particle villians Marian's There are two types of variances
1:34:44
One is called stochastic variants which means nothing but probably stick variants The other one is called deterministic variants which means readings That happens
1:35:01
for reasons that I know I know why that ratings happens Okay That is called deterministic But we're ings also happens in your data set For reasons that I don't know we call them noise Okay that is going nice
1:35:20
No if I have a plot like verse Alina model on have distribution of the data points around verse Okay for a given big for a given value of X Now look at this actual value Is this in your data Say but the model is saying it is this Similarly there's another actual value Which is this Why is this variant suckering I don't know I don't know why that radiance is looking This is called I don't want to use the words right now Abuse it down the line This variance that you see
1:35:54
for a given value fix your multiple values of why but they're very close But still there is a variant in them that we're Insys called stochastic noise Okay for a given value of X The fact that value vice spread around its central bar This is deterministic
1:36:12
We know that there's going to be this way All right So many address the out liars outlines is something which is lying to stand two divisions away or 1.9 600 Division of E You're investing outlined Okay But you're not addressing the random variance which exist in the data points Britain leave 1.96 and divisions
1:36:35
those will cost a scatter of the data points across the morning Prediction issued line actually is a black data points between the actuals and line There is a gap That gap is my stochastic I'm not an unexplained evidence I don't know why you temples Yeah in that too little deterministic is I already know my y bar has despaired and my access to spread this spread is on civilians I already know it has the spread I know it will be like this only I have
1:37:17
no reason for worrying about that What Empowering a borders forgiven believe X wire the multiple values of way That's so that won't be noise Is it because of lack of expertise I think you're going experience No one on the right OK maybe when way have delighted
1:37:54
this dance is back when the vendor different variables actually interact with teachers at this school in Arat In your data said you can you can have trouble with this noise that can happen The noise make it cancelled out or the noise may get magnified The way
1:38:24
the position of me Like like I don't know what this place That's what I'm going towards Okay so I have different combination off em and see So I moved away from my original mathematical space into another mathematical space where the dimensions are the different slopes and different intersects different lines Okay each line gives him some it The formula for this terrorist is some off school letters which is nothing good variance across dealing when every other square
1:38:56
terms This is called quadratic equation In mathematics all quadratic equations they have property The properties called their convicts functions convex functions means when you plot them when implored them against their independent variables MNC they will always acquire a bowl shape three dimensional bowl Such functions which acquired this kind of a shape a bold shape in the three dimensions Such functions are called convicts Functions they're guaranteed toe have one
1:39:32
absolute minimal edible extractions one absolute minimum So for some combination off MNC I'll get the least ever that combination off MNC which gives me the least error is my best fit line So they'll Gautam will start from some random MNC So maybe this is the random MNC This is my random M and this is my random See when I use this I get a line But this almost quarters across the line is very high So it's point here is one line in the original mathematical space sand or is a mathematical space
1:40:11
between X and Y If this Osma el one maybe this is the real one It had some m and some see that MNC is this point This point represents this line This line at some some of square letters
1:40:26
the sum of squared Idris is this so You all right No kidding Uh let's more
1:40:50
know what this algorithm does is it starts from some random MNC and it uses a concept called greedy and isn't descent As the name suggests we have to descend Okay Granted isn't algorithm which internally makes you soft What is called partial derivatives Partial derivatives And this term partial literally is I'll tell you order this It's not rocket science It is d Viber dicks It's exactly the Viber DX What It does this for a small changing AM in this direction or this direction Photo small
1:41:38
engine in this direction on this direction from the first in the random in for a small genuine em in this direction DX How much the other drops or increases so many changes in this direction I'm going further up there increases divi increases whereas if I go in this direction I reduce my m my divine decreases So it notes down that I have to move in this direction But they were taken one Lee m Right now How would see so do the same thing would see if I increases See slightly here What
1:42:18
happens to mayor of If I see slightly here what happens to matter I'll go in that direction where my every decreases descent So using this concept of passion verities which is nothing but the Viber BX applied on individual dimensions one at a time One active time That's what's compassion It finds out it moves from this line to the next line toe the next line to the next line until it reaches this point At this point water happiness whichever direction you move in Beyond this
1:43:00
there is only going to go up Better go this side or that side It is not going to come down for them So it realizes that it has reached the global minimum This is your best This is a mystery Our district my ministry So it does not evaluate infinite lines It uses this grading descent concept on finds out for the next best fit line next next next next and reaches a blooming This gives a word putting you get If you get stuck in linear regression the other function being quadratic it guarantees
1:43:57
you will reach absolute minimum OK but in the process off jumping there is something a learning step Okay they're too far away from this soil Not I'll just tell you the concept which is used along with special directors
1:44:12
If the learning step the amount of change that you do in divine digs If the learning step is too high you might jump and oxalate You might jump the global minimal going the wrong side and keep oxygenating backward and forward in front looks So the learning step is adjusted as it keeps jumping down The learning steps become smaller and smaller The various ways of handling this No intoxication One of the basis as it jumped towards your objective The goal The steps become smaller and
1:44:46
smaller This algorithm scored bold glaber Al gore
1:44:54
which is part of your great in dissent One variant of great interest when your little deep learning this partial derivatives will become very important for you to understand how the neutral it looks work how the back propagation off errors happened those things and come into play Okay but there's still a lot of time before we reach there Hopefully I'll spend some time on this one day and but you have areas of their Albertans know which will come up We just apparently better than this
1:45:28
Okay we'll talk about those things down the line Uh shelling one all of you OK your questions are still part right The two questions What are those Two questions Wait wait
1:46:03
Why should they be No call inherited What happens in business school In reality that is one of the questions that came up I remember that Uh good So my strategy Okay let's move on We'll discuss those things in detail right Okay So keep in mind we're looking at this data points closely the distance between the actuals and the line The line is the predicted values for a given value of X The predictive value of why is the blue line the distance between the actual video shown or the black dot on the
1:46:39
line That distance is shown by yellow dash lines This distance is a measure enough Okay the objectivist scram off These editors All these other squared that's some should be minimized Okay It is not very difficult Concept flies once again So if I my leaner model drawn like this best fit lying the data points scattered around this You look at the data point from this angle When you do that you'll see a spread This variance This is going some off square letters This is nothing
1:47:22
but excitement is export X body here is you predicted links so formula remains sing That variance is called some off scoreless that variants We have too many mice The minimum variance is the better your modernise All right so let me explain this to you because you're going to come across these terms down the light So let me explain this to a slightly more detail We're looking at this morning
1:47:56
on the distribution of the points The rate points is given to you The blue lines are exporting Viber on Keep in mind your model The best fit line will always go through the point We're exporting by about meat
1:48:10
always Okay now look at this Don't worry about these terms It's all I mean not very difficult Oh Sponsored Police official Ace Yeah
1:48:28
No If I take the Redpoint Redpoint is actual point in my data set in your data said in your data frame there is a record with value of X and it's got this morning value Why Israel But the one But the model the best fit line say's the value should be a look Plato
1:48:51
the distance between B two and B one
1:48:55
This is what we call enter the variance This is what we call some of scholars
1:49:01
Okay I'm sure you only one data point here But you have to imagine that a multiple data points here Sum of squared error place all together Okay so this contributes to some off square There's another term that we use This is called Total Later Total later across all the data points to take The total letter is nothing but the difference between vibe are the expected value of why on the actual value for you in your data there's a district distance between the actual value
1:49:36
of why and the predicted value the expected value of Y expected values Weiberg That distance is called total later off the total it of your model has captured this much Your model isn't immigration moral Hence the name given to this is the regression off The total error your mortal is predicted this much other
1:50:06
off the total every new data set Your model is predicted this much the difference between predicted under labour This distance is called regulation
1:50:19
What it has not being able to explain is this This is called some offs creditors
1:50:25
some off school Aaron has another term which is called Rested Doolittle's residues are those things which we don't have a reason for We don't know why they they're called residuals Some of screenwriters is also known as a residual errors So many buildings models especially when we go to your deep any part in all things you'll come across these trials Total later is the distance between your actual data point and expected value Okay so it's average height
1:50:57
of this class is 5.5 The difference between your height and 5.5 is a total error
1:51:05
and I have built a model to predict height off a person in the class based on the characteristics of the person So your model predicts some value for you So the model predicts so high to be 5.3 your actual Haider six six minus 5.3 that is 0.7 point seven is the different distance between
1:51:32
0.7 is this distance your actual Heitman is predicted Hide right the descent But in your height and the class average that is a total It'll okay so you'll come across three terms taught a letter some of squealers and regression keeping my ignition aided and and this is the latest These two together is a total later and don't miss Take the word error Error means variants It means medians off the total variance In the data set I told you one part of the variances cards
1:52:13
by the deterministic The other part of the variances calls by the stochastic long deterministic something
1:52:22
non deterministic is your residue later Some of spirit I think in this case I'm sure you leave one point But you have to imagine all the points together because this SSE Esser there across all points put together What I'm saying is what contributes stresses are necessary That's what I'm sure So you have to imagine all the points that that becomes too cluttered That's what contributes stresses are what contributes traces See that's what I'm showing Yeah he stares I told her little Miss this I
1:53:12
want to minimize this unexplained there That is what my objectives that great in dissent works in finding the best fit line for you Their best fit line is that line where this terrorist minimize because I don't want a model which is not able to tell me why I want a model Ish tells me why This So you know I go to do the recession No Is that a good minister I know it's because civilians it's because of the natural radiance in the process Okay when you're running a process many things come
1:53:57
into play Factors come into play because of its values Change here under around the expected value that is okay with me But for a given Elif input their multiple values of Why that I don't understand So forgiving aloof in production we've only one value of why Why they're multiple ballots of way that I don't that part It's my unexplained that I want a model which minute unexplained interest minimized
1:54:23
I don't want a model where unexplained that it is very high What is the use of the more it will not predict anything
1:54:34
Okay You think even you you have one
1:54:40
So this is that Is that Peter's predictive value But here multiple values off the once spread it on the line That gap is unexplained That variances board because stochastic random billions
1:54:58
you don't know why it happens So basically the idea is to minimize SNC That's all The bottom line It's minimus ecstasy So increased isis are assisted issue That is what our objective It's all right Let's more so before you build the model your to evaluate each independent dimension and C wars are value Our value coefficient of correlation comes into play to help you identify good predictors You're in the target Okay Once have built the mortal I want to see how good the model is
1:55:40
how the level the model ISS for that views another metric that metric is called coefficient of determination Munition on this is represented as are square but before you jump to it I should caution you Our square is not always our Into our are is a coefficient of correlation so naturally to think that ask what will be our into our that is true one Lee Then you build a simple linear model between one damage in one target
1:56:27
It's only true at that point in multi dimensional multi variant analysis very have more than 23 dimensions You can see our Squire's ardent O R Okay so ask where is just a symbol which represents 11 metrical for your determination How much off the total variance in your Y has been explained by your model Keep in mind the total variance and why is consisting of two components deterministic non deterministic stochastic random How much of this total variance in your way has been explained by your
1:57:06
model That measure is called Coalition of Determination
1:57:15
obviously a model which captures explains maximal off Bavarians and why at least behind very small residue ALS That model is the best one
1:57:29
so our square it ranges between zero and one zero means your model is not able to explain anything absolutely useless One means your model is able to completely explain all the variance in your target There is no residue
1:57:49
Such mortals don't exist we can ever build Such models usually are really will be between zero and one The closer their toe won the bet of the modern lists Because uh it's in simple ian models It's a square term It can never be negative But keep in mind in Multivariate analysis it's not just a square It's something more than that So long this range between zero and one Are you all OK How much How much Mitt is in white It's a this is eight
1:58:25
That means your model is able to explain 80% of the total aliens in your wife
1:58:31
total variance which includes stochastic and deterministic So your Oscar should be asked close as possible to one your model Very good so this metric is used to evaluate the performance of the mortal while our is a metric which is used to evaluate the independent dimensions when you're building the morning geometrically Okay if you take one simple linear model off one X and one way Okay that's what I'm showing here So this is where your expert and Viber meat please look at this carefully This
1:59:10
is my wife Bar This is my expert
1:59:13
Okay so there's an actual data point Excitement is expertise this way I minus Y But is this that gives us an area rectangle
1:59:23
We saw that early Right That area represents the total brilliance in your data point The total variance in this data point the co variance in the data point because off X and boy this is that area The light grey area the light grey areas This it's behind this doctor You get the light Grady is this that light Grey area is a geometric representation off the total variance because off x and because of why in this data point off this total radiance for a given
1:59:59
value of X immortalised predicting a value of white with this So this predicted value Why minus y bought x x bar This dark area is the explain billions How much of this total variance has been explained Captured by your model r squared is the issue of the dark area to light area What you're seeing on the talk This is the residents unexplained Willian's So our Squire is again an issue We have their issues between the total variance in the data Points on the variance
2:00:43
explained by your model That ratio is scored are obviously the lesser the residuals unexplained variance the better of the modern This way Oh I think you innocent The problem is go back to our discussion on our where I told you If there is no relationship within these to the target and they live in variable you have to give get a perfect symmetrical distribution But the problem is getting a perfect see medical dissipation zero which means you're
2:01:30
going to get asymmetrical distribution the moment you have asymmetrical distribution Your are a little bigger than zero magnitude this way or that way
2:01:42
which means our square will be greater than zero are square It's not considered as a very good metric for evaluating the models because it is easily impacted by fluke relationship So we make use off another metric Their school adjusted Aspar which is nothing but our square minus fluke There is a formula for this How do you find out the flu content
2:02:17
which I'm going to avoid right now because it's too detailed to discuss In case you're interested Let me know I can share it with you Conceptual is easy to understand what the argument is Very lengthy We won't have time for that That's I'm going to ignore it straight So the metric that will make use off to evaluate our models is and Justin are square not R squared the beauty of a distant as quick ISS We're never in your model building exercise to predict your
2:02:56
target when you include useless variables variables which don't really have any significant impact on the target But the relationship is a statistical fluke When you use that useless variables our square will keep on going up and up and up It'll become it'll come closer and closer to one but a district Oscar will go down in decrease so venue young include useless variables in your model and just for our security will go down It'll reduce whereas our square is likely to go up and to
2:03:35
start square will go one leavin you include good variables in your models Good variables a variable Swiss relationship A significant strong The views are just gonna square for evaluating our mornings
2:03:53
One thing that you need to keep in mind constantly getting clear Any attributes that he used in your model building any attribute that attribute will always have both Two components Good
2:04:05
you are and fake are it'll have both the conference But in a good attributes The percentage of this will be very very high comfort of this in a poor attribute Useless attributes Person does this and this will be almost similar That is why when arguably is closed to say point for 0.0.6 part off it will be caused by fake are part off Will be real are homeless Israel Hamas is fake I don't know That's why it calls for further investigation before you decide to
2:04:47
use that in your model When artists close to 0.5 point 4.3 we don't know what how much of that is fake how much of that is really we don't know That's why you need to further investigate how explaining to the free market actually causing you something
2:05:14
Many use random function to sample from population
2:05:16
that I since they are discussion men ever to do sampling and your technique you use clustering layered sampling mixed sampling border Your sample will never be a to representation of the rial population will ever be so It's distributions of recycler different from the distributions in the population that stands in distribution is what will introduce fake Did I show you that video will assure your universe to be understood Locks continuously changing You know it's never started Various
2:05:56
Your sample is a snack short afraid of you Okay I'm just taking you through The concepts are not telling you why What is the formula for all these things They'll become too heavy for you right now But I believe in Chile Take you through in case that interested some exemptions that this linear regression models make We already discussed this assumption off linearity The relationship between the target and the independent variables are expected to be lenient Oh
2:06:35
that brings me to another point Please be careful when you're our value is close to zero What does it mean Okay I think that's what ended up saying there is no Lena relationship with an ex envoy Okay When our values close to zero bitten by an X that means there is no linear relationship That does not mean there is no relationship
2:07:08
For example if I give you a distribution like this this is X and this is why And this is parabolic There is no Lini relationship When you find our value for this decision come close to zero But they're related There is long linear relationships So whenever you do this analysis are released close to zero Go back and check your pair panels Airport Is the relationship nonunion If the relationship is nonlinear and you're building a linear model in that your d a mystic a fundamental
2:07:38
mistake Okay same way If you have a distribution like this are valuable comes to zero
2:07:50
But there is a relationship it in y in x a sine omega t example You dude scientists are you know what tedious So when our values close to zero that does not mean there is no relationship It means there is no linear relationship If you have such attributes in your does it then you may have to think twice before use clean importance So so Unless we know this pair will not come to know that we have this kind of morning It is it always adjustable for 100% if you ask me a
2:08:30
lot of the same pair Plot is the most important tool you have in your toolbox
2:08:35
This you should use to understand your data
2:08:39
I suppose they got something different models Maybe I only need modern Okay So land Now that brings me to a question What is linear and what is known Meaning Okay now look at this This is slightly funny on at the same time something which is actually serious This y in X Okay Mathematicians will see Suppose this is X This is why And the expression is bicycle two X squared
2:09:17
Mathematicians will say this is a nonlinear function
2:09:22
Data scientists call this linear function Okay What We can replace this with some value OK so why is it OK So I can convert this into a leaner and then believe their models on this So many bills Linear models Why equal toe m x plus c
2:09:41
The reason why they're Korean here models iss
2:09:46
these coefficients they're not raised to the proper
2:09:51
They're raised to the power one That's why they're called linear their linear in terms of coefficients If I have a model like this Weichel toe m X squared plus C for us it leaner Mahdi Okay so don't get confused with this A lot of bad revolutions This is not linear but for us for us indeed assigned psychic learn or are this is a leaner more Show us a drink
2:10:23
You'll see You'll see that if you do in our 10 PT case in order mpg case impeaches the target on your xsl weight of the car You see that the data points are scattered like this You can still use nothing Non linear model see matter goto non lean immortals are models Were these coefficients themselves are complex questions They're not simple question They're not simple ratios There's something like e race to poor I Tita and all this kind of things will come over here Wait
2:11:31
so you're allowed to try both build a linear and non we never see which generalizes
2:11:35
Finally what is articulo objectors It should have minimal bayous millions which gives it that minimal bias Lady in seven put that calm voice What we call on somebody techniques in production We never put one single model into play We all this put a multiple models grouped together in a play we always install multiple mornings particular our own Good morning That's okay Cameras facing you know you it's okay All right Okay But don't knock the camera off He is also sleeping Uh right So what
2:12:17
we do is have you seen this porn binary code or apathy Have you noticed that the audience never go wrong Though the audience individually may not have very high I Q But put together they rarely go wrong This concept is called wisdom of the crowd The same concept is used in Gaeta Science Also in reproduction Isar models We don't put one single modeling to play We always put a collection of more intimately now just going on somebody The algorithm assumes that very selenia relationship
2:12:47
between independent variables on the target It also assumes that there is no relationship between the independent variables that all algorithms do There's nothing particular about this algorithm Assumption of normality off editor distribution This algorithm assumes that the model that you build the best recline the best fit line would be such that the variance around the best fit line will have normal decision What do you mean by this If this is my best foot line and the
2:13:24
data points scattered it on this and when you look at it from this angle it lovebird normal decision This is not guaranteed Sometimes you indifferent the best fit line on the best fit line You might see distribution like this Mawr errors on the positive than on the negative This happens when have outliers in your data Six This algorithm all algorithms will go down on their knees when it comes to our class But this particular algorithm is prone Toe Oh Class If an Oakland somebody
2:14:03
new data said the best fit line will get pulled towards that and get pulled towards will gravitate towards out lives Okay so when that happens the best fit line that you get the m and see that you get that MNC may not be the right M and C for a production It's a model may perform Poor leader Okay So many buildup models linear models Any morning The first thing that you do is handled outlast first Okay
2:14:40
The data various ways off testing this out whether this is happening or not One of the ways of testing this else you do a scatter plot between the actual values off why and predicted values of way okay which in actual values off why and predicted values off Why since the data spread out on the best fit line there should be no correlation the distribution to be scattered like this
2:15:03
predictive value If I actually fight there should be no correlation between them When there's no correlation between actual and predicted that means of data is evenly distributed across this press Fiddling
2:15:17
Sometimes you might notice there's a trend in these errors There's a train to some kind of a train in the in the difference between actuals and protected their trains It means there is a trend in the residuals as the value of X increases the residues increasing magnitude When you see such prince in the residues that means you're there something not correct with the other data sets Maybe you're having outclass Maybe you're building models on non linear data sets Those kind of things will come
2:15:53
out I don't know whether we heard this word home Most capacity off errors almost a nasty had close capacity There are two things that will come a prison statistics What it means is as you built a model the others should not spread out like a cool
2:16:17
the spread off errors across this model for all the range that you have the spread should be motor Lissy uniforms Sometimes you will see that as the value of X increases for the best fit line the magnitude of the other increases The spirit is still normal
2:16:38
but the magnitude off terrorists residues for high relief X is different from the magnitude of errors for lower value fix They're independent variables X sex Things indicate hydro scholastic whom a scholastic The expectation is it will be home musket as Homo means homogeneous Okay so the residents respective want a range of X is will be similar across all the range assumption of independence off Harris I mean ever done in one recordist movie influenced by the other
2:17:15
than another record error Down on one dimension is no way influence But the error down another dimension The dimensions are independent of each other That exemption is what is usually violated
2:17:27
But still regard and build our models I found a very good water call site which discusses these errors I'm getting to the Ural here by any chance have done this Durbin Watson Test start district You don't statistics Right
2:17:46
The inferential and this thing Descriptor started six
2:17:50
inferential statistics older than Z scores It is
2:17:53
You're done All the fetus and all these things have done right Okay if you're done those things And if you remember those things maybe you would like to visit the site
2:18:03
A lot of information please expand your horizons Go beyond what we teach in the classics That is just not possible to teach everything in the class for the time limit So go beyond the whatever we teach expanding horizons
2:18:17
Yeah that is smart will give you a very firm sound footing into data science and you'll be able to face situations confidently So whenever I come across any Urals any articles I'll share those things with you on this Late advantages were still not out of the words There's still much more to come We're not It learned the full stuff but we'll do hands on and then we'll see the other stuff Advantageous is very simple to implement
2:18:48
and it's easier to interpret Rewarded the number of dimensions business So when I have a question like Michael M one X one plus in tow extra plus c I can give physical meeting toe this I can say for everyone Unit increasing X one How much white in just that is going by M one
2:19:09
cycle Physically interpret this I can give physical meaning toe this question You can't do this in all the cases You may not be able to do this supposed You're using neural networks You can't interpret those models You don't know what they are Get on black box
2:19:27
How we can convert this whatever this formless telling me I can map it dual English right I can tell you in the day to day language but it is telling you
2:19:37
I won't be able to do that If I use a random forest you're going to do ensemble Very well Do under forest New Forest is a black box model We don't know what exactly you're doing inside support Vector machine is a black forest We don't know what black forest Blackfoot is A pastry black box we don't know what it's doing inside right This model is a white books How close can bring it onto its knees This particular model is pronto outlines are linear models A pronto class decision tree is not prone
2:20:11
don't lives And but this is very important point Since the linear model goes through the point we're expert on Viber Meet the best treat lines The lines have alerted for you Go through vedo a bargain Viber meat If your expert in vibe are of not reliable
2:20:33
then your borderless sinful not very level So if I built a model on our dimension why and X we have This is a very large spread very large spring The expert is not reliable An expert is not reliable How Well the model which is going through the expert making use of this expert Hold the model beetle Right So the more time you spend in selecting your attributes to build models the higher the quality of the attributes that you used to build models the better your models will be And the last Baldry
2:21:11
Selena this is a problem not in regression This a problem in classifications Okay so if If you can usually need motive for classifications also So if you're having a distribution of classes like this okay and then this and you're trying to believe your classic fires you'll get probably a line like this as the best fixed line So I've always this triangle below This is circles and one trying lists Whereas on the other hand if you have made use off nonlinear stuff long linear models
2:21:46
then probably would have gotten a better result
2:21:49
Maybe you were guarding something like this Okay so if you use non really models it might do a better classifications than linear models
2:22:02
So this limitation is not in regulation It isn't classifications Let's do one hands on Okay The hangs on is not auto MPT data set artists as it is didn't given here It's the other one with a simplistic So we'll start with simpler ones the other one which are given to scar Okay carpets A linear regression I've taken the data set directly found the You see a site It's called imports Underscore imports hyphen $85 data This is the name of the data file Okay So
2:22:40
installed Auguring it I'm radically reading from there
2:22:46
So let me begin Mac Plot live is an instruction to the Jupiter notebook The plot Make all the plots in the notebook itself Sorry Jupiter's over I'm importing pond us Lump I on from psychical and linear model libraries I'm importing the linear regression you can have Linear classifier is also Then I'm reading the CSP file which is available in this You are It's a comma separated file on I've taken all these column names from the usually does it So that gives me the data
2:23:24
frame which I call card underscored If there's my do Definitely all of you Okay that this is like loading a date and your table selects Lookout to records Transpose is not a must but since the number of columns is huge I put a transport I'm seeing first two records record zero and record one
2:23:50
There is obviously missing missing Return some of the columns and they're string data types In other columns Fuel types gas string Did it
2:24:01
number of those two It's a string later Time All right No problem Go down Our objective is to predict the last column Given all these independent variables can we predict the price of the car Is there a relationship in price and all these variables That is what we want to find out What is that relationship But before you do that look at the data types Many of the data types are object Object means string machine learning algorithms cannot handle string data types They're to
2:24:36
be converted to numbers So what I'm doing here this I'm gonna cover it in the numbers But before they convert numbers I'm doing I'm dropping some of these columns The reason I'm dropping these columns is if you go and take a frequency count on these columns Fuel type for example or engine location for example Most of the records have only one value for this This is called low variance filter Look at all the data provided to you in the data You have columns Where
2:25:12
in the Collins there is not variants at all There is a gender column gender column all the datas about meals then that Colin is useless It's not going to give you any information Such such columns are called low variance columns Though they're supposed to have multiple values they have only single values A few hours we remove them from the analysis So when I just did this you I wanted to this I've manly broz the data file this excel sheet CS Wi Fi You can
2:25:53
actually see Let's do it Know where we're talking Let's do this Um what you do is go to the site on Let's download this detested control See Okay don't It's stupid
2:26:19
level courts Thank you Now we'll save this down and save it as ah well stated us Let's David is takes document and then we load it up on this cold Okay But I'm not stating it considering Vestal Okay
2:26:42
Done Let's open it Okay let's start Excel
2:26:57
Excellence Running into problems It's evil Oh okay Okay It's going to clash with their ah recorder I'm not going to open this Let's get on to this court But do this analysis you'll find that these columns are having low very inspectors their defeat to be kicked out right So then I want to do is I saw the car data types the column Later times most of the data types of object string types I dropped my useless columns Number of doors most of them are four doors then fuel type Most of them
2:27:36
are giving the field tapas gas engine location Most of the Miss Frank one or two is in the rivers so on so forth
2:27:51
So once I dropped them I'm still left with some object data type Number of cylinders is object on any other off dictator type No one did That is object types It was single really very dropping Which means that the current price of the car is not a function of that Its function of other variables Okay look at this Hey guys do you understand why the drop low variance filter
2:28:26
the reason vital drop low violence will raise the price is not influenced by this This particular column which is low very ins If the price was impacted about this column this is low variance Then all the prices should be fixed right Which means the variance in the prices is because of the other columns not this cold So that's why you drop it now The number of cylinders what I'm doing here is I'm converting all these into numbers So wherever I come across a strain
2:28:57
called one I replacing numerical born to a depressive to you should do this with care Okay You should do this with care Suppose you had a column called Gender and the gender was male and female Then I cannot do this blindly like this Because if I replace a gender see a male with zero and female with one Then I'm introducing an order in the column that stole orginal columns Zero is Listen one But there's no such relationship in the real world So if you have a gender called him then I will
2:29:32
use a technique on one heart According we're not in cording I'll use that technique to replace my strings into numbers That example I've given you an auto in PC data Sit
2:29:45
Okay you look at that later On the other hand if I have a column called Income Group where they have income groups like low middle high not there I can go and replace low income groups Zero medium income one high income for two Because the same order which exists being 01 to exist in that column also Okay so if you call him is an ordinary data type You can go and introduce order in your numerical values
2:30:14
If the car column is not ordinary gender column then you cannot blindly convert them into one into you have to resort to one hardcore in psychic learned there's a there's off facility function called label in quarter naval and quarter introduces order in your data So be careful when you're using that in suffusing label in quarter use for not quarters All right
2:30:41
It has about way I just way yeah way Oh so be careful when you converting your object data type in the numbers don't end up introducing any order which doesn't exist
2:31:31
So when you use uh one home according have you all seen one According any time
2:31:38
you're not seeing or not according Okay I'll tell you what one according is then we'll move on So you have a column Gender on the Gender column Has male on female Okay this I cannot use as it is in my modelling So what I do is I convert this into numerical data types All later types have been numerical So when I converting the numerical I use one According what will Happiness will replace this gender column with two columns Gender underscored em gender underscored If
2:32:20
Okay wait over the recorders and you'll have one here Zero here Very With the recorders Female You'll have one here Zero So this column will be dropped in your data set You'll be left with this call One and zero will be stored on the hardness or in the in Very very storing It asked numbers Last used in this way Way way
2:33:32
He doesn't have one Boarding still gets right No object So I'm dropping my these data types Okay On Dhere card data types Let's see again Borden Stroke is to love dignity States Okay here I'm converting my cylinders into an American get types and here I'm converting my data types into this thing Flute Okay guys all of you with me on the cylinders Cylinder point Okay After cylinder point What I've done here is based on the experience ahead in the other data said where they
2:34:13
were missing values in autumn Pretty detested You'll see missing values OK so what we do is if you come across any missing values replace it with man Man means not a number none values OK so what I'm doing here is car into school Dev dot Replace very become across with some question mark Replacement man But I didn't find any question marks in this data set The question marks were there in order Imperially does it Okay so we don't we Can I tell you Delete this line We don't need it Then at
2:34:47
this point what I'm doing is they don't question marks There we are Sorry Very questions there Correctly sorted here Right We saw it here Question Mark Say sorry Mr There are question marks here These question marks now would be replaced with not a number null values
2:35:05
Okay when you replace cushion works bitten AL values you can do numerical python operations on it and converting the new appropriate values Okay That's why we did this move on my gator types right now are after changing the data types here After dropping the columns here some of the data types are still object
2:35:30
I've just been string data types So what do you do here Is I first convert my cell in Toronto cut into new miracles Then I changed my data types here Look at this card in the school Death bore
2:35:46
I replaced doctor as type flute I replace Stroke Column asked I float So I'm changing on the data types to float because many of these columns let's see which column headed Kushner normalized losses normalised losses It's already a numerical calm I think Yes it is already a numerical column so I don't need to change the Virago It's only numerical columns I don't need to change it Rest of the columns Natural dictator types I'm turning into floor types until this point Is it OK Once
2:36:24
obtained The data type three numbers Numerical I know there are married missing values So the strategy I'm using here for missing values is replace the missing values off Price column with the million off the price column I won't explain this stuff to you So I have a have a data thing And in this data fame I have one called this calling this cold price in this colander Many Innis
2:37:01
any any Wherever there are question marks we replaced all of them with Ennis The question mark has been left behind by the data entry operator because when he was entering the data he did not have values for those columns Those records So every place question marks with any soldiers became really because no these have become any I can use the function filling What dysfunction does this If you see that Tucker's syntax what did Dennis find out The median off this color What our values
2:37:37
are available Using those values find out the median Replace the innings with a million off this car It is always say for three please missing values with any sort of missing the lives with million because means they're easily impacted by outliers More this Using classic categorical variables We're going running with float variables so we always replace mean or median but we don't have checked out lives All columns will have outliers mean get easily impacted about lives
2:38:15
So if replace these values would mean ongoing
2:38:19
replace remove the old clients then this will This will be completely incorrect Three plays with million millionaires not impacted voters right And the beauty off this ISS in psychic learned I can find median off Even an object called
2:38:39
The column is supposed to be numeric but because it has question marks the data frame is converting their objectivity But you can still go out and find the million off the cold Why Because median is physical Central value
2:38:53
mean is mathematical average It's a calculated central value You cannot find mean often object data type You can find million often objectivity right All the forgetting the difference That is where the difference between bite on and our comes in are a lot of first converting numerically value even to find millions Now how do you How do you address missing values OK how do you address missing values Depends on your analysis of the data and by the values of missing The values can be missing
2:39:34
because of some random reasons that we call it randomly missing values values which are random You don't see any patterns in them They're missing values in all records and all colon's randomly distributed Then we say that data is missing at random There's no particular reason for it Okay we're us They're can be situations where the values are missing because off some reason they're not because of randomness For example if you take income usually will find nobody
2:40:07
wants to disclose their income So I will find missing values in that income Call many off them So depending on what is the reason for that Listen this You'll applying different strategies if it is income column where the values are missing because of the nature of the column Then I might have toe build a sub model a model inside a mortar
2:40:34
where I predict the valleys off the income based on the other attributes available So my domain knowledge will come into play which will help me understand what is the other attributes that this income column is going to rely on depend on How can I guess it
2:40:50
So whenever I create such imputed values as we call them I'll create a new column
2:40:57
income imputed So I will not touch the original call I'll create a new column called Income Imputed where all the data available is copied as a tous including the missing values on the missing values are imputed from the other columns What is one started you if your data is missing it Randall then replacing the column with a million is the default strategy views Look at the spread of the date on that column If you see long thing tales Venus Notre Level It's a very
2:41:31
place with medium right There is a package available called mice It's available both in R and Python in Python It is available in another package called Stacks More Stats Mortal mice stands for multiple amputations True Chained equations money People amputations through chained equations Yes when you run this of vertical package on your data set it'll automatically try to replace the missing values in all the columns with predicted values Where predictions will be based on the other
2:42:38
columns well established the relationship will see which one A strong relationship going impaired So one columns missing value might be imputed multiple times from multiple calls He's going to read about this It's a very powerful package and something comes with power You have to be very careful in using it So read about this mice I'm not gonna cover it here but it's a very building package You should read about this And just to add to the
2:43:09
confusion apparently the mice implementation in Piketon is slightly different from mice implementation in our no they both call it mice consistent with the values So these are the tools that you should be a write off here to have in your portfolio when you start doing better sense products I like all of you down This shelling the one once I done this Then I go and do my basic statistics
2:43:40
Dave Describe Describe a descriptive statistics and you know how to interpret this all If you're a bit of this take any call the height of the car First thing you should look at this difference Beating meaning million The mean is 5 to 53.72 million is 54
2:44:01
Not a very big difference 53.72 and 54.10
2:44:06
Not a very huge difference to worry about
2:44:10
Look at the distance between the length of the left hand tail and right hand left and Taylor's minimum minus first Corbeil 47.8 Let's take this 48 48 52 That's around four units Four units Difference between this and this is again four units so it's going to be very symmetrical Belco The distribution is likely to be a symmetric belco for height All right I don't see any skew to worry about Look at the car Wait Same story Repeats a 2555 to 34 14 If there is any column where the mean and median
2:44:56
are very different then you might be having a skewed business It look at this one price No prices that target Look at the price literally If over nonsense Stein's let's see whether it's tons OK so this is Ah numerical via statistical way of analyzing data But instead you can do a pair plot impair plot I always prefer to have the diagonals isn't form off density graphs How do you get that You've any called the purple you give Diagonal kind is Katie E Colonel Density
2:45:36
estimates When you do this you get the plot like this Now let us spend some time on this Always keep in mind the output of the pair Port is a square matrix It has asked many rules as many columns it has Okay so this is Roo's versus rules competitive So we have normalized losses than we have real base We have lent so on so forth Water Dennis from Cardiff I've taken using index locator all Rose Call him 1 to 16 Yeah Column zero is mice Ah call the 1 to 16 How many of their here wolf 6789 10 11
2:46:27
12 13 14 15 16 I have not taken the price for him here So what we do here is instead of doing this you do this You do this just knock coming this out Okay S in this don't bear plot Car underscored the if Katie you know this we'll do this pair No but on this So let's let it plot Fortunately it's coming out first Yes sir Garnet Now I want to make sure my price column is reflected here We're going to predict the price Right Surprise columns I think Second lost
2:47:34
or something Cylinder Mpg It's last but one
2:47:51
Why is it not showing to me I'm not getting it once I probably didn't Card underscored If you are able to get the price column don't stand in the data set It's there in this price Bearable Got underscored You have described it's there in the studio
2:48:28
but do you guys are getting it All right All okay Let's look at that Okay The first thing that you need to do is when you do pair plot is first thing that you need to understand This first True and first columnist Same second row and second column This scene Hard ruined third columnist Same So on so forth Okay And when I do a comparison off a column with itself if I do a comparison off se MPD versus Mpg What's this MPC What kind of a plot will I get Scattered Block
2:49:06
It'll be a straight line There'll be no information here There's no spread nothing So trying to do a scare deplored for a column with itself is useless It's not going to give you any information So whenever it comes to that point where you are going to look at the column verses itself 40 of dinos into giving you scatter Plourde They have given you a density coat by default they give you His programs were changing too dense surgical that wanted of doing that is in our If you do this pair floor it
2:49:38
legal The diagnosis blank it leveled the diagnosis blanket won't show you anything there That's what happens in psychic planus They give you a distribution I leave it blank user to show the distribution on that particular column Okay so that is why diagonals you'll see the distributions How data is distributed on that particular call
2:50:00
Okay so let's look at the first diagnosed
2:50:05
in the diagnosis In my case first columnists and bowling Um honestly I don't know what's in bowling is Maybe Actually I've done that homework Next economists Real base wheelbase is basically ho Spread the front wheel in the back field the distance between them Okay we see a normal distribution except for higher values higher values There seems to be some kind of pale Doesn't look very some look at the length of the car again Almost a normal decision But we see some kind of overlap
2:50:40
here between the you see multiple Gazans here one behind the other may not be off a concern right now because they're all kind off overlapping Remember which was the column where we saw a perfect match The central values What is it called Bore height How it was a call This is the distribution of fight So in decision of height now look at this The distance bidding meaning million was almost very and the sales were also equally for spring We were expecting a normal distribution
2:51:15
But enforcer it's not normal What it is showing here is two of Gossens to normal Coz are overlapping each other Okay this is what we call the mix off Gossens go to the other columns Look at this one
2:51:36
This one has a long tail on the right side A long thing it also seems to have But these constants are ignitable They're very small compared to the other one Okay so look at the distributions on the on the distribution part on eternal columns Some concern but not too much to worry about You should never expect a political symmetrical normal co in the distributions All algorithms zoom that the data on the independent dimensions are distributed normally
2:52:11
Okay that's all they assume Apparently not so But still it's close to normal so I'm happy with it Excellent competition Issue that compression ratio You see here this is competition to issue and compression ratio What I'm seeing here is there are two different clusters in it
2:52:36
Two buckets in the data on that particular dimension Okay this is a clear indication of mix up of fossils When you build models especially linear models you should not mix Gossens
2:52:52
Every Gaussian has a particular character So what do you mean the Gazans Gazans mints distributions
2:52:58
So they found a particular dimension You have two different distributions You're talking about two different processes which are generated This data number in started sticks You're done a defeatist or is that normal test When do we say that two data sets belonged to the same process
2:53:21
When they're significantly overlapping each other The central values are not very far away from each other If the central values are very far away from each other likely that the two data sets have not come from the same source The two data sets have not come from the same source How can we include them in one single morning What is The Moreton model is predicting the behaviour of the source The process with generated the data right What happens when you use a mix up
2:53:50
of Gazans like this is you're going to fight Next bar expert is going to lie somewhere in between these two which neither represents this slaughter persons This but you're leaner model is going to go through this bar expert
2:54:05
So you're linear model Rely very good for this prediction nor will be good for this production he has done some type O Hence a lot of typos sitting on the extreme end This happens then you measure this terms in some fractions on his other added one extra zero or something of that sort is done because of it using outlast clattered together in one group So I have to go in and find out what are these records which is causing this long tail in this Leicester My hunch and experience tells me this
2:54:42
is most likely a type of rather than data coming from two different sources Yes it is going put uh 1.9 600 division limit
2:55:07
Yeah this old list out less so I can detect anything which is lying beyond 1.9 extended division on this dimension So any replace those applies with values proper values then you standard division will collapse I look at this
2:55:45
guys since you this point has come up on this Does this in your original raw data Said suppose it's distributed beautifully Yeah applies here You do something about the appliance you identify the records on replace all these values with the median Suppose you do that Okay When you do that you remove all class replace all the values of million Your car will become sharper when the car becomes sharp Wolf Standard division decreases when standard division decreases
2:56:19
The data points with their earlier not outclass Now they become oh close because their standard division is decreased So 1.9 600 divisions both sides has decreased The new data points will promote us out Last which earlier were Lord I Did the brothers outlets on This will continue to go on infinitely So just handle the first level off out liars Forget the second level of old lives They're not really hopeless Yes 1.96 because $1.96.95 percent of the
2:56:59
confidence some people used to standard division someone believe in your three standard division That depends on your there It's a matter of life and death and I want my model to be perfect to ask those It's possible that meantime I want my deviations to be very tight where I can tall relieve some mirrors My divisions can be so it depends on your equipment because I'm two different clusters which I'm seeing here But since the size is very small My hunch is it's the whole
2:57:38
place on doubtless have been introduced through some
2:57:43
final problems you didn't Removing record is always the last option because they're going to lose so many other column informations Many removals records
2:58:09
We're only his genital here dropping the rules
2:58:12
So if you start dropping the rules for all columns their outlets your data set my shrink except one Everybody has outlined really moving out there for everyone 5 15 Dropping records is always the last option When you have plenty of in data science itself is restricted Dropping records is not a good option Shalem One all of you Okay let's move So no always start by analysing the documents first how the data is distributed in each column This is my univerity analysis This is what
2:59:07
the output of Deve describe us How your data is distributed on that particular column It's a basic status is a descriptive statistics Next we do is buy varied analysis in Bible it analysis how these columns interact with each other For example if you look at this one this is telling you interaction between the symbolizing sim bowling and this one will base
2:59:33
By the way in this square metrics about the Dagnall and below the bagman its mirror image Either we focus below the diagonal or about the date Okay so let's take below the diagonal So I know this first column is in bowling Seam bowling was Israel based What it means is as the value for sim balling increases you see that the real based trend is going down We'll base is shrinking as in bowling increases Okay but I'm also seeing a lot of noise for a given value of symbolic There are multiple values
3:00:11
off wheelbase So between Sim bawling and real based there is a negative relationship There is a negative relationship Okay look at this This is between symbolic and length of the car
3:00:28
Since length off the car and real days are almost kind of related you'll see a negative relationship here also but not as well as in this case Look at the relationship between some bawling and Victor It's almost circular distribution No significant correlation between the two All right we can continue to do this analysis Let me just look at look at this one This is competition reissue Yeah on some bowling for increasing the lifts in bowling compression ratio that you find is in this range
3:01:09
for one group off cars whereas for another group of cars the Sim bowling range of some balling is very small for this cluster
3:01:21
If the questions aerial cluster the range of possible symbolic values is much less than this question That's what is coming out of this analysis Okay now let us look at the other columns Let's look at this one Look at this This is rich relationship between will based on length Obviously as your real businesses will base as a real base increases the length of the car increases and you're seeing a positive relationship here The are really will be very close to seven or eight So
3:02:01
this is a case where you have two dimensions on both The dimensions are interacting with each other They're not linearly independent but the algorithm assumes that all dimensions are linearly independent off one another Okay so we have two ways of handling this one we're finding this is find out using your domain expertise whether calculating the length of the car is more prone to Arabs or whether calculating the real basis more prone to errors Measurement errors If
3:02:35
calculating the length off the car is more prone to measurement Arrest drop the length of the car keep the other one However if you're not able to take that call you bite You might want run a principal competent and Alison's a mathematical technique Our singular value decomposition another Take me Okay Using those techniques we can create synthetic dimension Or of this
3:03:02
that synthetic dimension will be a composite of these two soil dropped both off them and keep the synthetic damage I'll talk about this when we come to PC principal comedy Right now I like ignorance So in the data set what I'm saying is there are distributions amongst columns which have linear relationship There are distributions amongst columns which are indicating no relationship This one this one These are all no relationship but this is negative relationship Negative relationships
3:03:35
on It's a mix Some columns are related Some columns are not related Yes there's only one This is what does it over here
3:04:11
You will get the cars with number of cylinders Okay in your data said most of the cars have four cylinders Faisal under 608 cylinder Okay most of the cards Look at this Have four sitting this by the way these data points might just sitting on top of one another so that doesn't mean your data centers one Live 123456 records or four cylinders Don't do that Mistake Your data set might have hundreds of records for four cylinders but they're all sitting on top plays each
3:04:41
other's That's what you're seeing Only six darts
3:04:45
Most of the data have four cylinders Some of them have Faisal in there Some of them are sick some of them of eight
3:04:53
One lee Few records have 12 cylinders Very few records have two cylinders now Many go to the extreme end were cylinder bores the cylinder That's why you see a peak in full Some peak in five Daniel six and small P Care None of the other have any peaks Now cylinder is a numerical calm indigent of column You can have one cylinder to senator than what you mean by the spread What is the spread How can you have a spread on indigent columns This diagram which applauded here is called Katie E kernel
3:05:32
density estimates So there are three words in this First word is it's an estimate It's an estimation off the possible distribution density distribution density estimate in the population in the population How the cars are distributed around this value in cylinder it is a density estimates based on a mathematical function That mathematical function is called caramel So using a mathematical function on the available data I'm estimating how the density decision off this columnist likely to be in
3:06:10
population That is why you're getting the spread The spread is not in your data The spread is the estimate off the distribution density in the published based on the observations in your details So for integral columns also you can have spread like this because what that cardinal is telling us in the population it's like you could be like this more morning e I actually went and started watching Bowling is but you you can drop it You can't do mortal building without domain knowledge You
3:06:56
have to have dominance were to go and find aboard Disemboweling stands languished Ready I mean before more if a car coming from Japan but this car coming from us the mileage Mpg is going to be different so it's a categorical variable Origin of the car is categoric variable It has an impact on my break The way you look at the ratio of the distribution in a column there are eight values but majority of the records is that value one very few personal value to
3:07:56
then that becomes ah low variance filter That's a candidate for MoveOn's but the skin is in the distribution supposed to have Gender column Male female 95% or 90% of the data's males fighters in this females So I might take a call that this column is going to be useless for my analysis and drop it so it's a call that you have to take It may not be 100 0 It might be some decision but is that decision good enough for you to use it You want to make a call and that
3:08:26
call you'll people to take both based on domain knowledge If I know the drivers in the people who use the car it's 50 50 amongst these gender in the market I know that but for some reason the data which have does not reflect that this this'll
3:08:54
analysis is done on the original data said it's not on the drop data You should always the this analysis on Cardiff You should always do this pair plot analysis on the original data set Always Okay once Because this this analysis vote is going to tell you which dimensions like you to be important which are not likely to be important It always do this on the origin You got coffee
3:09:25
E No no no I change your Children
3:09:28
I change it here I asked Teoh I commented on this leg I just did that
3:09:51
You're talking about these Yep Okay yeah You should be doing this before you do this
3:09:59
before you always do your analysis on the raw data Okay Let's wait What you can do is go back to your data find out what is a car model going to refer to the manufacturing because you have all this information Whether this car model can have torn cylinders That analysis will do that That is what you're to do See Missing Really replacement okla replacement All these things you cannot do We thought your any domain knowledge You have to have the middle of this When
3:10:57
I went and studied this due to sit I found out that for a given car seat or it'll some some some particular with intuitive on particular civilian The data given for mileage was under liable because the factory manufacturing label itself says the mileage mpg is going to be in this range Whereas these guys claiming to be so I so the deer time till become a question mark So you have to do that And that's okay So when you do a real life project when you get into Capstone Project you have to
3:11:28
reflect all these things How did the analyzer columns How did you handle your old class How did handle missing values I knew to justify your strategy Why you did that You can't blindly replace something with a million That may not be the optimal strategy Oh this fact that risk factors sport can you just read out more loathe Okay All right So this is a risk factor associated with the car on I again this reminds me um the risk factor associated with the car depends on the color of the car I don't
3:12:23
think they have color here It depends on how the car has been driven how the car is driven Um these days a lot of people you know But they take pride in telling that my car has so many chips If by this high end foreign ground they all come with embedded chips those embedded chips in real time the capture the data a border driving style and pass it on to a central server may have they say down and analyzed And the risk factories registered recalculated recalibrated Based on how the Dr how
3:12:55
the car is being driven The symbolic reflects that Right Okay let's move Now are you able to understand how to interpret this pair plot They come to the experience but this is where you have to do it Okay
3:13:10
And moved on one So what I'm doing here this from the car in the school day If I'm dropping the price because that's my target And whatever columns I'm left with I'm capturing that into X So I'm creating a new data frame called X which along have only those columns which are independent variables
3:13:35
The target column price will not be part of fix then Once again I'm going toe I'm dropping number of cylinders here because this is uh the same problem It's ah no variance filter again But if you wish you can keep it Then what I'm doing here is from card underscored if I'm dropping the price Andi Sorry I'm not dropping the price I'm selecting the price column and storing it And why Why were able so X variable is all columns except price Why variable It's one Lee the price called So if you
3:14:14
have 1000 recalled 10 columns this will become 1000 record nine columns This will become 1000 records One column Are you okay with this
3:14:25
So we have segregated the independent and the dependent Very If this was our I don't need to do this because our takes in us input one lee one single data frame And along with that a formula there is in psychic learned are two separate this daytime toe independent and dependent variables Then I'm making is off the random function The generates the training set untested We call it uh train to split When I make use of the string to split on x and y I'm asking you to split the daytime the ratio
3:15:05
of 75 25 25 for test 75 For training the output of this will be 44
3:15:13
data sets So let me explain this to because some people will get confused So if my this was my data famed B if
3:15:26
this is my target variable ivy independent variables
3:15:30
target first thing which I did this I captured the independent variables and store them in X I took the target variable and started And why then on this X and boy I do that train test split When I do this what happens is it breaks this x and by both in tow X underscore tree x Underscore test Why underscore train Why underscore Test this training label by underscore train iss for this x underscore tree This is my test data except a score test corresponding
3:16:25
wine descriptors Now in a build immortal what I'll do is I'll call the fit function fit and that the fit function all supply These took this one x underscore train and why underscore tree I'll supply these two This is at this point the great in distant will come into play a lighting to fight for you The best fit model The best friend model will reflect the relationship with you know why this one on X This x comes from this trainings so it will be off the form Why is equal toe m
3:17:18
one x one plus m toe x two So on so forth How Maney columns are there 16 So ems 16 ex 16 plus c This will be the model Once the model is built we want evaluated We want to see how it performs toe this model I'll supply the extender school test values which means these values on it will give me the predicted values Wipe red out of getting it So I supply the test data The X underscore test one leader test later toe this mortal the respective Collins get multiplied by
3:18:13
their coefficients for every record for every record I get why in this corporate What is it predicted by price for this car in the data sick I know my actual finder score Testers This predicted when predicted by why is this What is the difference between these two that will remind accuracy A currency will be the difference between the UAE underscore test which is actually test labels versus wind The school spread which is your predicted test levels
3:19:06
Right So let's see this But before we go there Yeah well I think we can do it comfortably So let's look at this
3:19:15
train to split I'm calling the linear regression here This is where I'm in stun shooting The linear regression Morning This is where this class becomes on object and in the object that is a function called fit I'm calling that fit function supplying the brains It been around this This is the point Where the best fit line using grading descent This phone for you Now look at this I'm printing Are the coefficients off all those The best fit line The best fit plane hyper plane
3:19:52
brother So I bring to coefficients These are the coefficients Now we and we need to explore Understand what this is You said there 16 columns So the 16 columns of 16 different coefficients M one m two m 16
3:20:09
This is that m one m two m 16 m one is 88.57 m to 71.82
3:20:16
entries minus 93.38 All these are those M ones So let's understand what it means What it means this for everyone Increase in Sim Bowling One unit increase in symbolic The price of the car goes up by 88.57 units
3:20:41
De Viber idiots for everyone in it increase in the wheelbase The price of the car goes about 71.82 units Whatever the units maybe dollars yen rupees for everyone in it Increase in the length of the car The price drops by minus 93.3 units Now look at this This is where you need to be careful We know that wheelbase and length are positively related We saw that Then how come one has a positive relation Other has a negative relation but both of them should Boston
3:21:30
wheelbase and length are positively related but one of them is having a positive impact on the target The other one is having a negative impact on the target How can that be Such things indicate to you something is not right Energy does it And one thing which is not right in your data set here We all know that is multi Colin ality in size when Injun size increases by 100 the mpg a certain number of price price goes up by 2 51.74 units Okay that is very hard to interpret this But
3:22:07
what is happening is some of these ratios are kind off not jelling in there not fitting properly For example we already formed the length on the wheelbase They're going in opposite directions Okay that is one problem Then we have horsepower When horsepower increases by one or two cars car price goes down but it's actually even hospital increases by units God's praise should go up Horsepower increase means large cars large cost Tell you repay more place bit
3:22:45
more considered But this is not only negligible is on the wrong direction right Good point Right So there are problems in this coefficients which are not falling in line with our domain knowledge The reason why the 6000 is happening is probably have an outline on that dimension which is pulling it right So we're not addressed old lives here One reason could be that the other reason is if you're not ableto mean if these numbers are not falling in place Then we can convert them
3:23:43
in disease Course the entire data said can be converting disease schools on Then we can look at this but the magnitude of the difference you'll still see that the magnitude is still there OK the difference is still huge and the root cause off this problem that we're seeing root cause for this is one we're not handled out lives two we have some kind of ah vertical correlation between the independent variables which you noticed We're not done anything about that multi Colin charity So those
3:24:15
are the core reasons which will come across in all data set which lead to overall model level problems Okay not going to take you through further down into slightly more deeper stuff Uh let me see if I can cover it today If not we'll start with fresh minds tomorrow because I need that particular thing Needs a lot of brain energy So this cooperation on always have things then the conditions will not change When you normalize it doesn't like by normalizing you're just centering the
3:25:11
data to zero The relationship y in X will remain saying whether it's normalised data are no normalised Confidence will not change We need regulations are not impacted The normal additions they could is the school of the questions Everything from inside Okay Shelley Mom Yes This one was from him that this is simply need more If you build quadratic model then it will Right So let's more on Let's see the intercept This is the intercept I'm printing the interesting What you need to do is
3:25:55
model dot interested This is a modeling more lame not intersect This function this function will give it into set Whereas if you want to coefficients your to use mortal dot coefficients Look at this for coefficients It is uh Morton Dark wife Underscore Quit from the school Will print on conditions model dot Intercept will print the intercept Now the interesting that we're getting this minus 1 to 3 to three
3:26:24
dysentery in that area replacing the index The first one is intercept in the other Any of you have I just Thank you Yeah that is a defendant Nice isn't it I just think that Are you sure you're going zero and a i d X when you have to do this Say this is uh
3:27:04
this is a different any in that Very I'm accessing all the elements of their using this This is why it's printing out all this time You already done that after fitting it The output of this is best fit line the best fit Linus coefficients All the coefficients are internally store in that area which is accessible through this function This function is accessing that Eddie imprinting on the results Similarly this another very for your intercept It's accessing
3:27:39
that every using Ah interesting We don't know what that name off there is You can only access that every through this function Mm This point in the graph All these questions in the graph are the lowest point in your area This is area There are my table 16 dimensions over here This is the lowest point This combination of coefficients gives the lowest point in paragraph Best fit Right So let us look at the performance of the more along test data and as you can
3:28:25
see performance on the mortal in case there is 83.6% not back Not good either The comparison between this and this Okay so the 83.6% body thinking this No no no This is your coefficient of determination 83.6% of the total variance in your data Yes Data has been captured by your model are square It's not I just did not score this r squared E have a look at this is
3:29:20
exactly certainly wasn't oblivious to her because she has All right Intercept is one of these damages Editor is a function off is a function of slopes on C Error is a function off em and see Right So sees one of the dimensions What I showed you earlier was only one MNC Here you have 60 names and onesie We can visualize it so I'm not able to show it possible
3:30:08
It's the it's ah the coefficient of determining how much of the variance in test data has been captured by your model Explained about you Thelma one Okay No We need to improve the performance of the model Your first cut model will never give you the results that were expecting very rare So you'll always end up in another cycle Barrier object will be hardly improve The performance of this model How do you not Shut up you So there really is things you can do But before you do that I would like to
3:30:42
introduce you to a library calls tax mortal
3:30:48
stacks mortal The phone labia isn't if Stats Morton functions okay What happens is when the build is linear models in our our gives you a lot of start It's tickle information about your attributes on your models Psychic learned linear regression does not give you those information
3:31:11
but there was a lot of demand for that kind of are kind off analysis in Python So those people who support fight on secular and they came out with its library called Stacks Morning stacks mortal tries to replicate the behavior off our okay on gives you a similar kind of started surgical analysis as you would have received or gotten obtained in our So let's see what it does and why is it required So I'm importing from importing stats mortal dot form Lord Arabia and
3:31:41
importing some little SMF SMS is a bunch of status symbols functions and this I'm going to build a linear model using this But before I build a linear model using this have to combine my ex train Why trains into one data frame Because our takes as input one single data frame It does not take independent variables dependent variable separately So I'm recombining the two into one Using concurrent function
3:32:12
I'm creating a new data from called cars
3:32:15
which now has both the why on the independent variables put together for training Okay now look at this When you do cars head you get the price column in the first point First column and the rest of the boats Now I make use off the smf dot Well a soil It stands for ordinary least squares Linear regression Lena Regulation ordinary least square And as input toe this function I give price a formula There s a price is a function off the various columns the plus which you see here this plus is
3:33:03
a field separately It's not a mathematical plus
3:33:11
on at the end off it you'll see the data file name So the data file amis cars There you go Did I scars
3:33:24
So the output of this is once again the same model But this mortal has been built in our are based love are similar to our libraries So the model is price
3:33:45
is equal toe Ah I'll big The anticipated spin bowling is 88.57 88.57 Sim bowling plus Ah 71.82 will base minus I'll just take one more 93.38 garland seal and so on So forth The last thing being minus 123
3:34:18
23.6 or something Intercept This is your morning
3:34:25
This is that a presentation off your best fit line Now you can write according Java in Java where you have a function where function takes us input 16 variables Multiply those variables with the respective coefficients Subtract this intercept for that and give the output as predicted price Convert that Java according to run a bell jar file production Isaac your mortal is in production Well it so there is a more same model We Gordon Psychical and Seymour
3:35:02
really getting hit But here we're getting some extraordinary lives which we did not get in cycling Do some of these things look familiar to you Our square a distant are square Look at the difference between Oscar and I Just square I just wanna ask or you'll always go down Many are useless Columns are square will always go up So just a mascot It will usually be listed aspect F statistics analysis of aliens Remember these words Fisher statistics and other surveillance Then we'll also have
3:35:43
bse BSE What is easy A group you were covered in start Yes is a KKK information content Bs is based in information content
3:35:57
All right forget it No problem not required Let's go straight down Most important of these is here I just desperate I just ask what is lower than us 80% Is it okay with you You wanted you to decide
3:36:15
But before I go there I'll come down to this Okay I'm going to stop at this point What is the stable telling you I'll just take one example and you're to use that example on all this Okay Then I'll tell you why this is not there in python Now you take any any one of these dimensions a length The core referent for lenders minus 93.3 We already seen that
3:36:48
And then it gives the standard order devalues probably upheavals Remember your hypothesis Testing When do you reject Now lie prosthesis When you exceptional hypothesis you remember those concepts Remember Views A p value there right The values that is What is this OK now the thing is let's take one of these example Lend I told you to find the best fit line The algorithm will try out various lines okay and you'll find out the errors for each line for you using grid in distant And
3:37:27
I also told you that the best fit line that finds out will always go through the expert Beiber Oh goody So it would have tied Prado various lines for you before finding the best fit line for each line On this particular dimension length you can take any dimension or for each line on this particular dimension called Lent you would have 100 difference Looks He's a lame would have had a different slope different coefficient So those Korean Fishing's would have a distribution of their own
3:38:05
Those different coefficients which tried out on the length dimension before it found the best fit line Those questions would have had different distributions
3:38:14
off that distribution the for one given to you for best fit line minus 93.38 Maybe this one minus 93.38 How far away it is from the central value is doing by your Peace Corps This is miners one point toe Whatever this piece scores What This Peace Corps Peace Corps is Aziz for all If he remembers the schools examiners Expert Mr Under Division How many standard divisions of a data point is from the Central Valley View Z score for that Then you students to distribution
3:38:55
We call it peace scores Formless seem Remember Students three distribution Okay The formalist See So the selected value for question for land harmony Standard divisions RV It is from the Central Valley for the distribution of coefficients on land There is going with this the standard deviation off this distribution The standard division of this which we call standard error MP distribution that is given by this This is the standard deviation This is Harmony Z scores of a
3:39:32
It is your selected value is from the centre Lose this You want to play these two Multiply these two please When you play these two and tell me what is the value get into minus 1.24 73 you'll get back the same value You'll get back this value Okay Look at this If I'm doing Z score okay These core centralizes your data
3:40:15
converting this of the expert in vibrant zeros I take any value Excite Excite minus X bar by standard division is Muzi school agree
3:40:28
Expert is evil because it's enters the data So that means the score is nothing but excitable standard division So excite is nothing but these got into standard division here we call it be and we call this started dinner
3:40:49
So in a respective it started dinner It gives him thanked or the value that you had been over here Excite it gives him back they excite Okay so that is the way to interpret this off All the various coefficients tried out on this length Dimension did select you this one How far ever This from the central value If it is too far maybe it's not a level I'll do that analysis using this The most important analysis which I want to take you to is the people under What is the speed value
3:41:19
Tell you what the people is telling us I'm going to talk only for length but the same thing applies for all Want to speak values Telling us probably is point to
3:41:32
what is a cut off probably for your confidence level 94% 0.55% 0.2 is much higher than 0.5 What this is telling yours even if in the population there is low relationship between the length and the price of the car no relationship with relent and price of the cardinal hypothesis on your taken data from that population you're likely to find this kind of a relationship between lengthen price in your data It means this is a fluke relation
3:42:10
I repeat I want this People is telling this probably finding this kind of a relationship between lengthen price in your later even if it's there's no relationship in the population in population that the slope is zero between these two and from there you're taking a sample random sample The probably finding this kind of relationship is very high which means this relationship is fluke It's a statistical fluke It's not reliable Look at this one people You uh they look at this one stroke be better
3:42:50
Is almost zero What it means is if in the real world there is no relationship on board on the price and then from that population you have taken the sample finding this kind of relationship with board in price
3:43:09
The problem is very low Doublet is very low I'm talking stroke Sorry Probably This is very low Which means this relationship is not fluke It is likely to be there in the real world Okay so let me put this pictorially and explain this to you Then we'll wind up Let's take one of these columns Stroke and price Same thing applies to all It's supposed between stroke on price price It suppose there is no relationship between these two Then how will the scatter of the
3:43:45
data points Look I agree All of you are very will be very close to zero The scattered the data points bring circular Okay Now what you'll observe is most of the data points are near Expert by what I said Go away The density false That is why you get a plot like this on this on a plot like this on this
3:44:09
Are you all OK When I convert this into three dimension bearing the third dimension is my expert by bar If I plot the density on the Y axis question to you is how will there how in the distribution Look I don't know Cylinder Given my very close It's very high as a movie the density falls So this will be your normal co A three dimensional Lomonaco A three dimensional laudable curve is a shape off the density of distribution between two dimensions which don't correlate
3:44:54
This is the shape off null hypothesis Now hypothesis stays in your universe The data is distributed like this Agree We don't know what the universe looks like But now hypothesis states it looks like this If so on from there have taken a sample on in the sample I'm finding the distribution between this score Whatever this succour and price I'm finding the distribution like this This is export of a book I'm finding a squashed Belco I'm not finding a cemetery Belka I'm finding a squashed
3:45:34
bell cough This relationship is borders explosives this coefficient Now the question is what is the probably finding this kind of a distribution in your later If the reality is like this
3:45:52
that problem is expresses fever Lou If the penal Israeli law listen 0.5 that means your universe is not like this We reject in Aleppo Texas whereas if people is good and important Five We say that there is not sufficient evidence to reject in a hypothesis I'm unable to reject in Aleppo Texas may not be I don't know but I don't have sufficient evidence to claim that there is a correlation wear trying to establish whether in Aleppo this likely Retour not We can't do the
3:46:36
other way We can't prove alternate hypothesis Is a fusion ship already No Not like POTUS is no relationship there independent It's always negates your alternate alternatives There is a relationship now about There's nothing doing this No relationship no relation shipments This is a B Now the question is from this your drawing the sample in the sample You're saying this What is it Probably seeing this kind of redistribution if it's coming from this If the people is
3:47:06
listen 0.5 we reject in Aleppo Texas If people lose points verify or greater than equal toe we say we don't have sufficient evidence to reject the null hypothesis So the exceptional hypothesis likely to be true We reject this
3:47:25
This analysis are provides by phone doesn't And if you look at this this is the P values So this tells you the importance of that tributes Those attributes to speed values is getting the points of the fight Useless
3:47:40
started circle Fluke Those attributes were people This is less than 0.5 Good attributes Let me be a killjoy Okay In the statistics community in the real world there's a vertical split between the statisticians One school of thought say spiritless is not reliable The other school of start which comes from the conventional status to export a few They say people lose a reliable The reason why they say people is not reliable issue can go and check it
3:48:11
yourselves Suppose I dropped one of these dimensions with high p value sim bawling and renew this analysis All the other P values will change Now the question is between your price on your length If there is a relationship
3:48:32
given by the coefficient why should that coefficient change when I include another dimension Live it
3:48:39
Why should I change It should not change right The relationships should remain relationship why should the intensity of the relationship change when I add another dimension to it But when you actually go in analyzes dropped the dimensions you see that all coefficients change correspondingly All the people is a lot to change The reason why the change is calling charity They're not truly independent since there is calling charity and the impact each other The confessions are not
3:49:08
reliable Hence the people is that not It is because of this reason this split between the statistics community on whether they should rely on this a lot psychical and guys say it for there that we don't need this
3:49:22
we have other ways of establishing The lab will be off the dimensions and the model select use that That is why a psychic learned does not give you this facility But then subsequently under pressure they came out with stacks mortal libraries which gives it this Okay now just to end of the show what is telling you speak value on what it is standing with the 95% confidence within which these values like Okay so if you guys remember is 95% confidence level area of rejection
3:49:53
area you'll see all you'll be able to connect to those incidents I'm going to jump this I'm gonna move forward Uh last one
3:50:03
overall Probably value off the model itself Stand Such a relationship between the price and the other good attributes Can't that's such a relationship Exists really is very low which means now hypothesis Overall it's rejected more than level people is very low It's after removing all the unwanted dimensions Hopefully you'll have a reliable model
3:50:36
if you're working for people who believe in started states And not that I don't believe in service Six Those people expect the values from you Then you have to use tax model if they don't care about people is don't come here There are the ways of designing with Dimension is reliable on summer techniques and all these techniques are available Have you used that he was decision greeting that a list out for you The dimension as you'll see now after lunch listed for your dimensions
3:51:09
in terms of importance Bitter dimensions that morning
3:51:15
did they use basic statistics They don't use P values euro tonight every business you don't have the knife is you remember busy and uh this thing distributions They use that for this Some starting positions say that basing distributions are motor level lives The name logistic regression is slightly amiss Normal It is useful classifications It cannot be used for regression Okay though the name says the regulation But there's a reason why it's called regression not logistic Classified
3:51:48
reason is under the hood Logistic regression uses linear models Leaner English That is the reason
3:51:59
it's a classification method based on a linear regression The response variable that is a target variable can be binary class before turn on the folder The verdict No nobody or it can be multi class classification Also I can use logistic regression to for optical character recognition
3:52:21
I can do that I'm in my personal experience I have seen and I have also read some papers about it Many compared Morten's
3:52:30
often logistic regression comes out on the top
3:52:34
if not exactly in the top At least off you top two or three Okay four classifications So I'm going to start by showing you buying a reclassification But this confused for multi class classifications Also given your independent variables
3:52:56
what is the class likely to be That is objective off this model I can also use a function available in psychic land we're insert defining which class and observation belongs to our test record belongs to I can print out What is the probability that this record belongs to distrust verses this close I can even print probably piece Okay This particular function is nothing unique to this It's available in all probably defense models predict you don't love
3:53:28
us Predict we just add Underscore Prabha this function will print out Problem develops if you go for just predict this imprint all the classes Decision Tree also has this No nine days algorithm also has this logistical predict underscore problem Let's take a hypothetical situation Don't worry about the property Let's take a situation and discuss this concept Andi for that we are going to use three colors You are a data scientist you're working in a bank on you have been taxed with you have been
3:54:28
given a responsibility to build a mortal where the border will take certain inputs on the output of the model will be Is this person likely to default if I give him loan Is this person likely to be a default or wonderful too You have to build that model Okay so what kind of data will you need to build such models Stream comes off score OK seven stories and more direction So let's keeps ableto your need Has a general income Any crossbones yesterday any probably here as ones of so many things Well
3:55:17
this is you That one of the things given to you as has England Forget it This is a touchy subject So let's call it H Okay One of the top Another inputs given to use the age of So you have the past data with you The bank has given your past data Sit in the data set You have both Before it was known before us and one of the columns in the data set his age You notice that you notice a pattern here The pattern is most of the defaulters Okay Most not all Most of the defaulters I didn't
3:55:55
lower age bracket Okay And you also notice
3:55:59
most of the known defaulters are concentrating higher age bracket But that doesn't mean there are no non default Isn't Loveridge okay You are observed This pattern less since there are two classes here What we're going to do is we're going to see Green Class We're going to sign a numerical value one great class numerical value zero Remember we can't work on string that addicts So the target columns will be assigned a numerical types Okay Why I want to this voice zero that No reason
3:56:36
you can flip it also fish But we assign a value of zero and one we don't assign any other value were signed 10 and one There is a reason from it
3:56:47
Okay now you're your object Now you find that the pattern over here As income increases
3:56:58
you find more greens As income decreases you find more reds You're gonna linear model already So what you do is you're trying to build a linear model here So you say Let the Y axis be probability off belonging to any one of the classes Let's call it one The Y axis is probability of belonging to class one you can actually take It s probably belonging to class zero Also if you like the two probabilities will be in worse off each of it So you can pick and even you like only so
3:57:39
they have been the leaner model The lean immortal Anybody is a black pen here Black Black No no This been okay Now you look this that as the age increases if this is a probably the probably off somebody being belonging to green the probably increases linearly
3:58:07
probably increases I can represent a linear models
3:58:13
If I have taken this as it'll if this zero then this line how would it look It will be reversed That's the difference So you can use any line Okay So I'm taking this as one for ease of explanation because you will be able to connect to what we did yesterday So the line will look something like this So this lying why equal toe wise property of belonging to blue class is related toe M X excesses Variable plus cc's The inter sectorial semen busy does even been negative But the problem with
3:58:55
the lean in mortal iss linear models go to infinity minus Yes Plus probably these have to be thin Zero and one I'd be below zero Horrible one So we can do Selenia The model views another technique There we transform this line into an esco This Esko is scored a sigmoid and it is very easy to achieve this The sigmoid is nothing but one by one plus e ease Oiler constant views and mathematics miners MX plus c So this best fit line The best fit line which is formed for you This breakfast
3:59:49
line is fading toe this transformation this mathematical formula that isn't of this transformation Is the school the property Is it read Zeroed infinitely
4:00:06
It'll reach one that infinite That's exactly the program Beautiful program No let us interpret Want this formalizing Keep in mind we're building the models of which means we're in trainings it So in trainings that we already know the labels off every record right The algorithm gets bored The input variables and output variables X and y So all the labels are already known on the labels are shown in the color or years Are you all ok Shall proceed now What The mortars Now that I
4:00:43
got sigmoid I'll remove this This line I'll remove this line Okay This is going to infinity You don't even need this We don't even need this Know what This Ah modern is telling you next Look at that If you think this rate point Look at this If you take this rate point what the model is saying is the probability of belonging to blue class What is the property of this guy Suppose we don't know the color
4:01:25
What is probably at this point belonged to blue class It's very low which means it belongs to the red class So the model prediction and actuals are matching here The mortal prediction on the actual label Both of them are matching Take this green one one the model is saying is what is a property That this brain point belongs to blue class This is probably blew class right The problem is very high and he's actually green So both prediction on actuals match Hey eight leaving
4:02:06
Are you with me Sure 100% Okay Don't get left behind your expressions Tell me a lot of stories I've wanted to move there and I lost count more than 250 stations no in data science So I built a model You know what's coming right So looking at that place the position people occupy in the class I can guess lot of things about the person believe me the old full of patterns and your you'll be surprised and notice this pattern without fail Based on the position people are occupying the class Okay of
4:02:45
course Late comers They don't have a choice But people who come early okay Based on the positions they occupy I can draw a lot off conclusions inferences from right So people who are feeling doubtful but are able to I asked their fear They're not asking I know I cannot infer such peoples so feel free to ask Okay All of you please feel free to us Don't get left behind I was always a backbencher I was always a member of the backbenches Sadie I'm not saying anything to you Don't look at me
4:03:23
if you if you sit in all places means I was always the backbenches so I know Moreover the back bench then the friend print But other patterns which are observed was doing decisions Right So a deviation from our main topic of discussion Bottom Linus Don't hesitate to ask Okay Feel feel free to ask Don't respect coming back to this Are you all with me on this tour Discussions Okay No Look at this point You see a great point here This one bought the mortal is thing is aspirants income he belongs to
4:04:04
Probably if he belonging to blue classes very high So the model we've posthumous blue Sorry in disguise Green green in this case green So what the models are faces telling is predicting the problem of belonging to the green class It's very high So the modernist glass of famous green But he's actually read the model is done An error over here Mistake
4:04:29
Similarly if you look at this one this side the Greenpoint here Can you tell me what the model is telling you The modernist earlier The problem If in belonging Toa green class is very low So he's likely to be read but actually is green So the model is done in error here A mystic
4:04:52
Okay So whenever you have overlapping data sets
4:05:00
as in this case in spite of the best fit line the classification you'll find some mirrors They will always bears thes error Circle training is errors in the training dinner I like no problem Uh which one is minus one This is Eagle These little this is one on this axis In this particular picture this access is probably have belonged to a class war Okay that's why you're esque of is coming like this This is come has been built out of a leaner model If we had taken this if you're taking this
4:05:45
as probably belonged to class zero then it will be reversed this cup because they're covered by like this The line would be like that Oh yeah All more than separate without Let's every answer the question Why is way
4:06:14
way or fitter Who said that In linear models we don't have any boundary They go to plus infinity minus infinity We don't want that Probably is have to be between zero and one That's where they were converting this ignored sigmoid curve Has that property that low vis remained between zero and one severe were able to map the numerical values to probably function Yes In multi class classification it'll be one versus rest Suppose I used to think
4:06:52
this largest integration for OCR optical character recognition
4:06:56
So you'll have one esque of for a worse is others one esseker for B versus others so and so forth So in your mathematical space you'll have my people s curves cutting each other One is for a vs others otherwise be vessels of their services and so on So forth All these algorithms Decision tree naib a support vector machines noodle It works modesty They all break your mathematical space into compartments the break them into compartments pockets
4:07:32
where one pocket belongs to this object The other pocket belongs to that All of them
4:07:38
You're done Decision to yesterday which doesnt do you use That is the human evidence Dennis It Okay That is a binary classification actually So when your big decision trees Okay this is a structure to have a physician trees
4:07:58
What you're doing here is this entire mathematical is Please is your root Lord The entire data said all data points put together a sea route node Then you find out a one particular column on which you break the date under two That is like putting a vertical boundary here So these two no are these childhoods But you have not achieved your uh article home virginity Your Gini index is still very high or in trophy is very high It's still not pure leaf So what you do is you again find a column
4:08:36
here in this data set on break this into two That means you're drawing a vertical boundary So this on this is a child not here in here So decision tree Also British mathematical space into pockets So start each pocket becomes homogeneous at least level So right now I have reached the leaf level Hopefully at leaf level Each pocket has become homogeneous
4:09:06
Now I know what are the conditions under which somebody belongs to this pocket That is not the rules of the differently All algorithms break your mathematical space into pockets including logistic
4:09:23
You green day s guys One for this is so my name is class specification If you're in multi classless taken optical character recognition
4:09:34
then you treat At one point in time a old data belonged to class A alphabet A s one and rest of the more zeros See what is the boundary Then treat all the records belonging to Class B as one and rest of them zeroes Then read all the boundary solo Multiple boundaries Cutting each other A person fits paper surface but multiple hyper surfaces for a for B for C
4:10:19
Yes yes the whole modern Really big conference it off All the Salafists Okay lets more someone big lens Always one was a Syria It's multi class classification Most of the algorithms give you a We are over here means one person's rest from algorithms Ah give you a facility off replacing the O v e R with basing properties Okay you can actually go and modify that in your hyper parameters
4:10:52
But most of the algorithm by default value year some algorithms like support vector machines over here is not very efficient So to give you another technique on base in probably based
4:11:05
someone Okay It has a grading descent algorithm one working under the hood to find out the best fit line All right The linear mortal uses grating disintegrate from B remember Great interesting discussion we had yesterday see the great in descendants to find best with final quickly Summarize this I know it was too much in one day to cover your error function will always have this form Not very difficult guys If I give you equation Y equals X square for any value of X Y
4:11:36
will always be positive It's a squared if exist minus one minus one in minus one becomes plus one This is called a convex function So whenever you have any function race too poor to it'll always be convicts And I know this a single dimension This extended multiple dimensions So if I take two dimensions M and C and floored the error this will be a bullshit Right now in this ball shape we have reached the global minimal It's guaranteed you will have only one global
4:12:12
minimum You won't have any for quadratic error expressions You'll all this have on global minimum guaranteed Okay so we tried to start with some random MNC that random MNC gives us some error the same concept of place and logistic because logistic is also leaning more right from night MNC It'll come down to this using your partial de Ritter's concept Okay there's a lot to discuss in this I'm not going to discuss all these things at this point but you leave this if you're going
4:12:49
to do deep learning neural networks and all the things at that time will take a dip in detail Same concept will apply today in logistics because largest accuses linear model under the hood So it finds tell you the best fit line given the spread off the data points mathematics space or best fit plane or hyper plane Once that plane is fouled
4:13:12
that plane is sent through the signore transformation
4:13:15
and it's converted Briscoe Right now the driving force behind this here in this case it was quadratic In this case the S curve It is formed This scum can be shifted forward backward upward onwards Let me see if I can show this to you Okay Um
4:13:46
what happens This insert If I take this esco I hope it books it works Okay now this is cool You can move it
4:14:03
up by changing the sea in Weichel Damn explicit You can move it down You can flatten it to flatten it What do you think will use the slope em I am off the line By changing the slope I can make it practical flattened or move it up and down using C So by using these two parameters M and C I'll find out which signal it's Her face I get is the best surface and the find out which is the best signal surface off all the infinite possible peace It uses a function That function is called log loss This is
4:14:49
just like a great in distant Okay don't worry about this Function is actually very easy Looks dangerous but actually it's very easy Now let us look at the long list Function looks very dangerous but it's actually simple Very simple Okay so forgive the signal This things just late Pick the expression Why I into log off Pia Okay Plus then one minus Y I into log off one minus y I into long off one minus beer I predict this Look at this expression What is this Why Why is that not a good
4:15:29
video You know the target variable ranges from zero and one Only one is for green
4:15:39
on zero is forward So this way I can take a value of 01 Okay now let us look at four situations I'm going to show you four cases on the four cases we're going to discuss here where is the four cases there These are the four cases Look at the first case in your data The first test case These are four test records Okay The first test record has this income The model is saying that probably belongs to blue Colas is very high and is actually blue class Noida zero it right
4:16:29
literate Come to this What is the Y value for blue class One into what is a probably very high Almost close to one
4:16:40
Log off a very high number When law comes close to one this is Eagle log off Very high number zero close to the
4:16:51
to raise to poor zero It's one There is locked of the base to Okay It's a lock to the base to off a very high number High number means close to one We'll always be zero Come to this tomb This becomes one minus one We don't even need to evaluate because this is zero
4:17:11
So when you do correct classifications for the blue points your editor is evil Let's take the next case The next cases This rate point Let's take this reformed Let's take the case Number three or let's say case number two The second case okay Asked for the model given his income This the property of him belonged to blue Class is very low
4:17:42
which means mortal is predicated He belongs to the red class Ask for the income group off this person The model the model is a blue line it's saying probably a fin belonging to blue classes Very low Anything below 0.5 is very low So the model is predicting that this guy's going to be read
4:18:04
but actually is blue So let's see what the question says His blue values One probably is very low that he belongs to blue class The moralizing probably is very low Log off A very small number is a very large number So this is into a large number Okay since is a blue class this expression You don't even need to revalue it It'll be zero so renovated of misclassification of the blue you get a very large area
4:18:43
All right So the more the blue class and is on this side larger the it'll become So in technical parlance what we say is more confident This algorithm is about a particular classifications which we know is wrong More their evidence Right Go down So for this blue point where the problem tiffin belonging to blue class is almost zero this it'll be very high Wait Come to the will first discuss only the blue class The more blue points are on this side In this case
4:19:45
the more the errors will come out larger There will be okay Now look at this The red one The fourth one are letting the third ones are found One we know that this guy is a before the classes right The model is given the income The models predicting is going to be different Come to the C Christian What is the label value for this guy Class rate zero So this expression become zero We don't even need to elevate Come to this guy One Montecito is one into log off A very a
4:20:27
very small number number minus one becomes almost one So this expression turns out to be zero Log off Very large number Okay so zero So this expression comes to close to zero for rent classifications We used this part of the equation for blue classifications to use this part of the question Take the last case where the red class is misclassified The model is saying he belongs to blue class
4:21:10
Yeah See the morals their faces properties belonging to blue place Exactly Look at the last case Last cases the fourth case where the fourth case is misplaced Eviction of red So when you do misclassification off read this expression is not for red So figured it Come to this This is one minus zero into a lot of the base to This is a very high probability that you model is predicting so this becomes a very small number
4:21:46
long off a small But it's a very large number Say area is very large That's on the sea Creationists So some of the others done across all the data points those which have correctly classified those which are not correctly classified Sum it up You'll get the total error across the intermodal Some off squared editors same greatness So the object is to minimize the sum of squared arrest by finding the right logistics surface Given the classes yesterday
4:22:40
the some most predators were driving the Reaganism here This in private Sigmar What happened together Actually what happened together It's on one after the other It wasn't You know what Thank you All that person It will happen when you're all the Big Seven Castor is doing all the new seven artistry What condition is even All right shell more extension off leaner More than that we saw yesterday The beauty of flow Distinct Englishness Uh it makes no assumption about the decision of classes in the
4:23:45
feature space Many of these algorithms linear model Especially if you're building linear classifier Zalina regulation They expect Gaussian distributions You understand the film Gaussian distribution all of you No Okay When you build anymore expectation ISS if you take this attribute the data will be spread around the central Valley of this attributes with some standard divisions saying that this attribute will have a distribution with Sandra Many applaud this together
4:24:19
together in the mathematical space they will look like if they're independent off each other They look like this We already discussed this This kind of her distribution is called ghost So in three d it looks like a bell curve Only this algorithm does not make any such exemptions But keep in mind if there are oak liars in your data set Even this algorithm will fall on its face because it is based on linear model So those Oakland's have to be addressed What will happen
4:24:54
if you have vocalize If you have no close you like you toe have doubtless for say for example vertical imam the class one
4:25:03
If an extreme outlined here then you're in trouble if I wrote light on this site No problem for Oakland on this side some more extreme It is most severe outlines as more the address will be right Okay you can do multi class classification using either by normal distribution in this case are mighty normal distribution Also it's possible you can also print on the property values if you're not interested in the class But I want the property values What is It probably belongs to this
4:25:44
class of that class quick to learn because it's based on linear model and you have grading descent to help us out to find the best fit line quickly fast in your tests did also President Old So it's very fast Andi As I said many times I have personally seen lots of strict progression stands out among the top few in many of the situations resistant to a war footing Because it's a simple linear model leaner mortals cannot be very complex shapes it is resistant War
4:26:22
footing assistant or fitting does not mean resistant Tobias serves It means this instant variances But we have another one to deal with And you can print out the coefficients the property values and interpret the property values So the model is not a black box right Disadvantaging classifications is it uses linear boundaries What it means is if the distribution of the classes
4:27:00
if the distribution of the classes is something like this when the model will suffer Right now we took a linearly separable this thing But what if the distribution is not linearly separable then logistic regression will suffer If the distribution was like this Mitt overlap But it was like this then it'll look well But if the discretion like this on that body decided you see the difference The difference iss
4:27:38
If I plot the density graph Okay In this case the tour dense recalls are overlapping
4:27:48
Your pee test and analysis of variance and all these things will will tell you that this this record belongs to disclose that class 50 50 it won't be able to classify
4:27:58
On the other hand if I look at this situation you're overlap is there But the central values are shifted apart So when we have distributions which are linearly separable may not be 100% Then logistic will do well But you have distributions like this I don't know whether you noticed PMR didn't do this It is like this That is my MP murder Does it You'll not be able to get accuracy beyond 70% 36% Do whatever you can
4:28:30
right It will not give you beyond Santa Fe when the data distributions like this none of the algorithms Very well 100% year lawyer does Before you start building you're to know your data which means you have to know every attributes How the data says distributed If you're in classifications how the classes are distributed
4:28:58
which dimensions or which attributes are able to legally separate the two classes Use one lead those for your classifications models You have some attributes They're both The classes are significantly overlapping including the central values Using such attributes for building model is useless and reminding you'll see all the dimensions selected What if we could
4:29:21
be doing more than four Who What human bring cannot do an algorithm So you have to have at least one attribute As you will notice in another data set Very useful bank dot CS We they're also their data overlaps on all the dimensions between the two classes But there are a few dimensions two or three with the data central values and separated apart Based on this tree it gives a 96% accuracy I think this can no You find out how the density of the plot changes When you move from this direction
4:30:18
to this direction you'll see that the density is very low very low very low increases very high highest Then it starts falling So when you convert this into a three d it becomes a normal distribution in freely This becomes a novel classes for city the access for analysis In this case I'm switching taxes or analysis which means say where this becomes excessive So that what is basically saying this No I'm not I'm this way I'm comparing these two with this This access is more
4:30:53
I'm just guessing if you have a distribution like this Your logistic will do well if you're a distribution like this the largest people suffer any Albert OK but what do you thing is Country switched to this access where the two classes are separated Or that's what you should be doing So if you have to access to attributes on 100 boot there lot linearly separable at all But on the other attribute barely nearly separable used other words
4:31:32
So use that which is having your own business Random forest Every Lord of the forest
4:31:51
is constructed using randomly selected features from the data set Okay Yeah it hasn't really attentively does that But random for this is the one living which comes close towards us No it doesn't Picks up doesn't even a visa
4:32:16
Okay let's move Let's do a hands on Ah this is the same date If it reminds There it is Same deal Say shall we do hands on on this Okay all against Are you already Okay So if you have access to the court the first line is backcourt lib We already know what it means I'm importing from psychic learned linear model logistic regression Yesterday we had imported linear regression
4:32:49
Okay then I'm importing back Florida Spirit PC born I use Yvonne libraries Former plots especially pair float comes out very well in this I'm going to use this train display ideology No water This nobody told by phone And to measure the accuracy of the model I'm going to use metrics So whenever you're doing classifications on of the most important tool for evaluating immortal is your confession metrics Okay so that comes through this metric slavery We'll see
4:33:20
all those things don't like earlier this female deputies used to be available You see I data set but now I think it has been removed or archived So it's not available But I downloaded it from kagle dot com Another site I'm lording that they doesn't Over here It's a data file It's a text file Why Luring the data file I'm feeling the colony inside which I got from the original source The colonies So the corner names are the first column limits pregnancy How many
4:33:54
times the weather way this particular test was done Animals Pema start rides a tribe in South America where apparently the ladies are prone to type two diabetes very early image So this research was done by this Union City where the optically waas can we build a model and predict who's going toe Have Type two diabetes by the teenage Okay for that they did the research I'll on the community Some of them were diabetic Some of them were no diabetic and the character sticks that
4:34:29
were captured Elise The first one is number of times pregnancy reported by the subject Next oneness plus more level in your body plus minus a component of the blood Then we have blood pressure This ah systolic It's systolic That's diastolic blood pressure This one's destiny that's telling depression Then we have skin Were apparently people suffering from Type two diabetes in that tried They have some kind off skin Four months This is mission went back Then they
4:35:02
do some testing but this is called fasting Sugar test some things that is this test
4:35:09
Body market index police Another test Some pedigree test They do I think it's another test related to blush over age On the last one is a target called whether this person was diabolical nor debuting Diabetic means one The Lord number Tickets zero Now the whole thing that the objective is we're trying to find out Is there a relationship between the independent variables and the target variable What is the relationship you're trying to find out That relationship
4:35:40
we assume exists in the universe in the real world we're trying to find out whether we can discover that relationship So for that we're building this models Onda I always do this based on the experience on God impeded is it When you look for this head okay and look for some sample records you see all numerical values But what is the guarantee that there is no record further down which has some no numerical value sitting the numerically Kahlo's What is the guarantee How do
4:36:13
you identify that Try to find that we can use We can use this method What I'm doing is I'm using a numerical python function Israel This function is a binary function It's a boolean function and you through our faults this function I'm applying it on to look at this apply I'm applying it on toe on the rules All the columns of this This is the beauty of pie tone where you have to do minimal according If you have to do this in Java you'll probably write a four next within a form
4:36:57
extort here So all the things taken kid off So what's happening is I'm going to apply this function Israel on all the rules all the columns of people live Okay on you see the still loaded This is a not operation I want one lead Those records very returns off falls Okay Fortunately there are no records where it returns the false which means all records All columns are new miracles
4:37:31
What is the other way off Finding out whether any such no numerical characters have come into the data You're like leaving next That's it Dave Types when you do return their data types for the columns if there is any non numerical value in any of these columns it'll return it US object That is easier than this gift we're going I didn't further record convert that into any any in new medical Then you can replace a medium mean motives replacement studies here We don't have
4:38:12
any such case right now okay But we don't have nothing to be happy about because we're seeing a lot of zeros on zeros in blood pressure is surprising It can't be One possible zero in plasma is not possible but they have zero values and blood pressure You have zero alleys and plasma so missing values are there The missing values will cost problems We need to are just amazing Okay let's more do it If you describe how the data is distributed on the Vegas dimensions
4:38:45
uni variant analysis If you don't like this don't do it Do the pair plot Bear plot also gives you visually the same university analysis Okay So quickly tell you one look at the difference between for the test data
4:39:04
the sort of the test attribute Look at the difference But in the mean 80 almost 80 are in the million 30 The mean is drastically shifted away from the median on the higher side which means there is going to be some long tails on the right side on the higher side being that's easily impacted about liars So some old cloud is pulling me towards the right side How do you confirm that Look at the distance between the third quartile and this Look at this
4:39:42
curry Look at your like me you don't want to calculate Can you tell me how long is long The right side They look at the land and comfort that on the left side till we left saying there is no tail 719 units So is this possible or is this a typo We need to go in check Maybe this is a typo We don't know Okay right I'm going to move on The next thing that here to do is first understand how the data is distributed on your various dimensions If you think there are outliers we need to undersell place
4:40:23
We already know we have missing values We need to adjust missing values The next thing you need to do is since you're in classification how many records are available for each class in the data set Look at that
4:40:37
The number of cases for non divert EG zero is 500 Where is the number of cases for diabetic is half off That almost off You're preparing for the aboard exempts And most of the questions that your soul practice on is calculus A small part You do it on trigonometry Obviously you're going to flunk in trigonometry right Why Because a pattern of questions that come to you in the exams in trigonometry may not have been reflected by this patterns that you have used which means
4:41:15
you're not going through all the possible patterns in the trigonometry questions Same thing will happen here to the model I have given when the to 68 of the diabetic cases Maybe this to 68 does not reflect all possible permutations that lead to devotees So the modern will perform poorly in predicting the diabetic cases will perform relatively well in predicting the non diabetic cases But I want the rivers The object is the rivers Okay And keep in
4:41:47
mind whenever you're in classifications on the classes are skewed like this All algorithms are biased towards They hire represented class all models The objective is to minimize the overall misclassification so their focus will be on the higher class
4:42:09
So this is another source off bias which comes into the model where the algorithm stencil Surprised So how do you handle this Make fun to do way Call the strategy up Sampling Don't stumbling Okay there's a started T call up sampling down sampling which I'll refer to which I'll take you through when we do eff MP mortal tuning But right now we're not going to do that There is a package available It's called England but right now we're going to live with this There's
4:42:46
another very off improving the same classes The classifications performing better in the unrepresented class that is called mortifying fresh holes Officials in logistic regression nine days Any other probably based model the cut off is 50% If the property of belonged to Gluck Blue classes 51% that the data point will be classified as blue
4:43:16
If the problem of belonging to the blue classes 49% the data point will be classic for this raid So the boundary is access people so Okay I can modify the boundary though none of the algorithms allowed me to modify the born free directly I can build rapper clothes and on the algorithms I built a model around the model I write my own class where I modified the threshold Okay once again uncovered this in f empty Not know if you twitch so we can play around with the social modify the specials to
4:43:51
improve the accuracy of the underrepresented Plus we can do that Okay How do we do that We'll take it little But whenever you see in classifications killed distributions like this be really this Normally you can achieve a very high accuracy for the underrepresented Plus unless you play around with these two techniques let's move So this is my prayer plot Uh okay I don't like this so let me quickly read of this All right I'm going to come in tow I'm not going to do
4:44:32
all this I'm gonna come straight to this Where I'm going to say it's in a spare plot Comma bag Underscore kind is equal to Katie Colonel Density Estimate Since these are numerical values it makes sense to have this on Dhue Human skull A is equal to
4:44:52
the last one What We call it Target Plus So Hugh is equal toe class Okay
4:45:10
You just get on the road It'll take some time That's mastectomy Yeah There you go
4:45:31
No Let us understand this That brought all if you have it Next Look at the very falls to Dimension The first dimension here is pregnancy reported Okay The blue is known deputy close and the red one is the deputy class oranges that are diverted class Okay if you look at it the two distributions are overlapping with the central values They're overlapping very small separation Or here come to the next dimension The next dimension is your plasma
4:46:13
plasma You see long pains on the left side with a small bump This is their old class the zero values But otherwise both the classes the blue classes perfectly looking looks like almost like a normal co And the central values are overlapping The central values are eclipsed So once again not a very good attribute But at least we have some differentiated here Look at this one Even the Oakland roller This is your let's worship These are the zero blood pressure cases They're also left
4:46:57
Yeah So there's no over your blood Pressure will be able to separate our based on blood pressure You cannot distinguish diverted quizzes longevity
4:47:05
There's no those What are you taking We have like the same niggling so both of them will shift together All right so we have a problem there Look at the next one This is even more surprising Almost gone Skin color skin not skin color This is skin fold Uh you see the same problem heroines every dimension you look at it Every dimension has a strong are none off the dimensions except for a few dimensions where you see smallest difference in the two classes None
4:47:51
of the dimensions are separating the two classes out So even you put all these dimensions together and build a model you're not gonna have in Yeah it's overlap Almost this case the previous this one And that's what you're seeing here The two classes are old lapping on every day They're not shifted Lingers Night Gordon Build a model Let's not be very hopeful about this Any mortal will give a problem here So what I'm doing here is this is not required It's not required to
4:48:32
be done All Al Gore from scan takers input the data frame itself but early rise To do this as a practice where I'm converting the data frame into Inari So data frame minus the column headers It's going in every okay on that Very I'm taking the independent attributes into X on dependent at a good boy and then I'm splitting this X and Y into train sickness it is This part will give it to people All of you with me on this all of you inside him Year olds on this Great Now
4:49:10
look at this I'm in Stan shooting the logistic regression morning You can call it anything you like I'm calling it more than I'm doing the fit This is where the best fit line is found and convert into your signal The fit Once the mortal has been phoned then I make use of the product function off that object on test So these are my predicted labels Diabetic nonverbally The actual labels are white Ist okay that some people get confused here so I'm going to stop it Don't look beyond this to stop it
4:49:50
I'm asking you to just look at this part Still respond Everything is fine Okay Now what I can do donate donuts I can For the time being I'll I'll remove this Okay I'll put a yeah for the timing Just ignore this 100 x Well put it over here Insert so below uh 40 Yeah All right Island committed Lee under that Okay Now look at this Have you seen confusion mattress before Yeah All If you agree it's exactly what it's called It confuses people a lot Confusion matrix So what I do is
4:50:52
as a practice when I make my confusion matrix I always keep the actuals on this side Actually Divert Take non diabetic on the predicted output of morning on the same diabetic None of this is my practice You can follow the other way but which are very you follow fall only one How do you control this since again White s first This is actual It goes to the rules my predict Next This is still predicted values goes into the columns If you flip this you'll flip back What it's saying is off the
4:51:35
147 diverted cases in your Sorry This is non diabetic Know that this is indeed non diabetic This is Deputy Andy the awfully Sorry
4:51:51
if you go and read the source of the data there very clearly told Zero is non diabetic One is deputy off the 1 47 known diabetic In your test data 132
4:52:03
have been classified as non diabetic No mistakes
4:52:09
for this class trip posters off the 147
4:52:13
15 have been identified as diverted incorrect classifications
4:52:19
These are those values off red lying on the extreme and valleys of blue lying on the extremist Look at the diabetic Plus Ah a 4 84 There are 84 diverted cases in your test data off this 84 46 have been correctly classified as damaging 84 minutes for distances 38 38 have been misclassified Okay now look at this Now look at this
4:52:58
What is the overall accuracy 77% looks looks decent but let's look at a crazy in double the class Okay We call it recall
4:53:14
Recall ISS the formula for recall iss through posters by true Boston's plus false negatives It looks very hi fi but what it means ISS two positives divided borough total So the recall ISS 46 divided by through monsters plus false negatives This is false negatives for this class This is false Negative right So yeah So it is for this explicitly Eight back to 84 That comes to around 55% Okay
4:53:53
It's coming by 55 54% Right That is like putting a coin heads or tails You don't need a model for this So to get this kind of accuracy you don't need a model You can just toss a coin and say Diverted No never That is not
4:54:12
so Order looks great But don't get fooled by this Water is important is this But this is what we expected because this is represented very poorly The damn ridiculous This is important for us Look at the recall for the non deputy class None of it class The recall is 1 32 by 1 47
4:54:36
This will come very high 90% 80% 90% roughly Don't the recall for the non deputy classes Very high recon for the deputy class is extremely poor But that's what you should expect We wanted the reverse So that's what will happen when you have situations like this
4:54:59
Okay there is no way you can get a very good model When the individual attributes are poor attributes on the classes are underrepresented
4:55:09
You can't do it So the magic happens in the data How do you prepare your data for analytics That is where the magic this it is not in the algorithms Okay
4:55:23
The difference under different 80% off our project Estimated effort in data science goes into preparing The data running the algorithm is not then arrests around 2015 to 20% goes into fine tuning a model It goes into that All right let's do Ah there is a function called score Inter doing this I can directly called score But don't rely on this Then you're doing classifications What dysfunction does is it regenerates model and dentists it on this test
4:56:01
data It's regional smarter than compass attested time this automatically and gives it the score OK so this is nothing to do with confusion metrics you can I tell around this and say you remove the hash it on it
4:56:20
It should give you the score mortal It's no different What's Billy mistake Well im not executive I'm just showing of this okay I have not executed these lights so it gives your own score But overall school it's not reliable so don't depend on this This brings us to the end of the story alone Digression Analysis on Before I sign off I'd like to inform that we have launched a completely free platform called Greek Learning Academy But you have access to free courses suggest the
4:56:57
iCloud in this still marketing to thank you very much for joining recession and have a great learning
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat