Skip to main content
Loading Explore guides…
Linear Regression · Acedra
Home
Explore
Computer Science
Linear Regression
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
360 segments
Copy
Hide video
0:00
Hey guys this is part of me from Crete learning and I welcome you all to the session on Lee New Regression So Linear regression is one of the simplest and most widely used algorithms and machine learning on Dive created the studio real in such a way that at the end of the session will have complete understanding awfully near aggression Now before we start for the session I'd also like to inform you guys that will be creating the city's off high quality tutorials on dear
0:25
science artificial intelligence and computer vision So please to subscribe to Greek Learning Syrup Channel and click on the by liking for that you have a notification off our upcoming videos Now let's have a quick glance said the agenda
0:44
We'll start The key study will understand the need of regression analysis Then we'll have a brief introduction to simple linear regression and multiple linear regression After will have a demo in both bison and our language is going ahead Will have Professor location was the academic director at Create Learning explaining all the concepts off union aggression comprehensively from some orchestra Has it worked 20 years off industry expedience and market
1:08
research project management and data science So let's start off with this key city Now let's say you are conducting a survey on a bunch of college students on You'd want to understand if there's a relationship between those CDP off the student and his or her g r E Score Now the Broden statement is basically this So you were understand of the students who have ah hire CDP all certain to have a higher G i t score And similarly if students have a lower cdp then would they have a lower G I
1:37
t score What is the problem Statement Now To find out the solution for any problem you need to gather data So doctors what You go ahead and do it So let's say you go to around 500 odd students on Ask them that Jari score and also ask them the CDP and then you create a table out of this But if you just look at the stable would you be able to draw any sort of inferences from this Obviously no So this is where you do some sort of data visualization So or here You've got this Jari score on you
2:05
got this CDPF Now what do you do Issue will map the g r E Score onto the Y axis and you'll nab the CDP onto the X axis and you'll build a scatter plot for these two on As soon as you build this graph you can clearly draw inferences So what you understand us As the CDP off the student increases the G R E school off the student would also increase So their store for positively new relationship between the CDP of the student and the jihadi score off the student So with
2:34
this you can clearly state that if a student has high cdp then he'll have Ah hi Jerry scored Similarly if the student has a low CDP then he'll sort of have a Luigi Ari scored Now this inference was quite easy to draw from this visualization Now what if I actually changed the question So let's say if I asked what would be the geology scored off a student if his c g P s 8.32 now if you just look at this graph or here on def I see that the CD piece 8.32
3:04
would be actually a bill to tell what Toby is Johnny score Now that would be difficult And this is exactly where regression analysis comes in So let's understand what exactly is regression analysis So as it has stated over here regression analysis basically helps us to understand the relationship between two or more variables by creating a predictive model Now this simply means that you have a dependent variable and you have the independent variable So as we saw
3:31
in the previous graph we had mapped the jihadi scored onto the Y axis and the CDP onto the X axis But when it comes to regression analysis will do something different So in regression analysis you basically have a dependent variable on one or more independent variables Now we will map this G R E score as are dependent credible and will map the CDP as our independent variable And we will try to understand How would the jihadi score very with Inspector the CDP or in
3:57
other words we obviously trying to understand How would the dependent variable change with respect to the independent variable Now since we already know the value of CDP which was equal to 8.32 So I would have to take this value off CDP plug it into the X variable and then I'll get the value off White with his out regression analysis works Now the simplest category off regression analysis is Lena Regression where we basically try to understand the
4:23
lean your relationship between the dependent variable and one or more independent variables And the important thing to note in linear regression ist you're dependent Variable needs to be a continuous numerical value So and this is what we have with respect to what problem Statement So oh here Johnny score is our dependent variable which is a continuous numerical value And we are trying to understand How does it change with respect to the CDP off the student which
4:49
would be our independent variable And when it comes to linear regression we basically have a street line So a straight line is a linear equation and that is why this algorithm is known as linear regression So we have why and we're trying to understand how does why very with eggs So this be what you see So this is the coefficient off X and this is basically this loop off the line And this a what You see this is the intercept off the line on the Y axis Now let's accept this concept
5:17
ofa linear regression to the previous problem statement So we had the jihadi school of the student map to the Y axis and we had the CDP off the student map to the X axis Now we know the CDP of the syringe was 8.32 So let's see if this value here is 8.32 then the corresponding value off the G R E score of the students would be around 308 all 309 But the question over here which arises How do you know that this is the best freight line Now let's see if I draw another line like this And the CCP
5:51
of the student is again 8.32 So for this value off 8.3 to the G r E School of the student could be something girls So this time the jihadi score could be around 312 or three Island 13 So as you see when we changed the equation off the line over here the value off Why also changes Let's say I draw another line like this so again the value off excess seem but the value off by will change So this time though Jerry scored would be around our 305 No Similarly we could
6:21
have a 1,000,000 Are the lines passing through these set off points So party know which is the best fit line right So to understand that will take this example here So there is something less measure off good fit Now instead of taking all of those points we'll just take the set off six points forward here and I'll start by drawing an arbitrary line over here now these green points which you see So these are the actual absurd values for the value of X So when the value off excess equal due to
6:52
the absurd value off by this five But then again the value which is being predicted by this line would be around 2.0 so their difference between this absurd value off by and their predictive value off why by this line is what is known as the residual or the editor in prediction and similarly you'll have a residual for all of these points over here Let's take this point Now hear the X value is equal due to on the absurd value off Why s four but the predictor value off Why would be around
7:22
3.8 Similarly if you take this point x value as four and the absurd value off by a stew But the predictor value off by over here is five And if we take this point X value is five absorbed value off Why seven and the prick Their value off wires around the 6.5 right to these dotted red lines that you see these basically indicate the residuals Although editor in prediction and get that best fit line don't you agree that we would basically have to reduce
7:52
these individuals as much as possible And this world we have something no other residual some off square So what we do is we'll go to this first residual and we'll square this up Then we'll go this second residual We'll square this up go to the third residual and then we'll square this up and we'll do this for all of the residuals and then we'll take the summer off all of the residuals Now when we dig the sum this has to be as Louis possible And whichever line would have the bluest value
8:20
off residuals are more squares That would be the best front line Now let's extend this concept of the previous problems statement So this was our problem statement where the jihadi school was on the Y axis and the CDP of the student was under X axis And we've got these three lines forward here So for the first line the residual sum of squares comes to be around 28 for the second line The residual thermo squares comes to be around 22 And for the third line
8:46
though this duel Samo Squares comes to be 24 Now when we compare all of these three residual sum of squares the second line would be the best fit line for our problem statement So now we've got the best fit line on We only know that or linear line will have an e question associated with it which is basically of I equals MX plus C And that is the question over here So here why is basically the jihadi school of the Student on X is the CDP off the student and these two
9:15
which you see the OM value and the sea value This we've got from the best freight line So they em value which is the slope of the line comes to be around 6.8 and the Y intercept comes to be to 61 Now to get the G I T School of the student All we have to do with black in the CDP off the students which we already know so will plug in 8.32 into this equation and will get a final value off 317 So this is how you were able to get the g r e score off the student by just knowing the CDP off the student
9:45
And this is how linear regression algorithm books
9:49
Now that was a simple linear regression where we had one dependent variable and just one independent variable Now will extend this concept and do something known as multiple linear regression So in multiple linear regression will have multiple independent variables instead of just one independent variable So to understand Dr will take this data set a war here So this is the famous empty cars D does it where we have these columns So this is the MPG column Then we've got the weight of the car
10:18
the horse part off the card the number of cylinders in the car in the number of girls in the cars And we want to understand How does the mileage off the car very with respect to all of these other independent favorables And since mileage is our dependent variable will denote it with y and all of the other independent variables would be denoted like this So wait I'm devoting it with x one horse par with X two number of cylinders with extra and number of girls with export on This is the final
10:47
equation off this hyper plane over here So it is typing Why equals X one plus bx to play CX three plus DX four plus e And this would be the equation off the hyper plane So implementing linear regression algorithm will be working with this diamond's use it on Let's have a glance of the description of this Does it So the studio said comprise off 10 columns and there are 53,940 does Or in other words there are 53,000 940 diamonds So we've got the price
11:18
column which basically notes the prices of diamond and US dollars Then we've got carried which you know it's the weight of the diamond Then we've got God which is for the quality of the cut to have got out of these different columns over here Then we've got X y and Zed which basically you know the length of the diamond the birth of the diamond and the depth of the diamond and millimeters Right now before we go ahead and start of by implementing the linear
11:42
regression algorithm let's do some dealer manipulation on dinner visualization and let's understand this she does it properly So while started by looking at the structure of the street has it and to look at the structure of the data set I use this str command And inside this ill passengers did acid which is diamonds
11:59
So I'll type and diamonds over here Right So these are all of the columns over here So you see that characters on your medical column and then we've got cut color and quality which are factors Then again we've got a new medical columns over here Right So this is the structure of this So we saw the structure of the data set Now we'll go ahead and perform some data manipulation operations So let's say I wanted one year subsidy off this entire day does it where the cut off the diamond is equal
12:26
to premium and the prize is greater than $5000 So for that I'll be using good reply package So let me load of the deep lapac Ege So I'll type in library off the Flyer on but we've got the filter function from the Deploy a package so I'll start by giving the name of the data set which is diamonds on Then I'll use this pipe operator Now this pipe operator is used to connect different objects and functions together So with the help of this pipe operator I am connecting this diamond's data set
12:57
with the fate of function and inside the filter function I'll give in my conditions so the first condition would be the cut off The diamond needs to be equal to premium
13:07
so I'll type in premium over here and then I'll give in the second condition which is the price of the diamond needs to be greater than 5000 So I'll said the condition which is priced this greater than 5000
13:20
and I'll sew this in our new data set on name It does Let's see a premium price Let me have a glance of this So I have a glance of the deal I said I'll be using the view function And inside this I will give in the name of the data set which is premium price Right So are of the 53,000
13:40
diamonds we have extracted only 4717 entries where the cart off the diamond as equal to premium and the price of the diamond is greater than $5000 Right If you look at these two columns closely then you see that in the CART column The value is only premium and in the price column it is greater than $5000 now Similarly let's do some other operations Now let's say I want the length off their diamond to be greater than four millimeters so let's go ahead and given
14:12
that command again I will start a bag during the name off the data set After that I will use the pipe operator and then use the filter function Onda Inside this I'll get the condition so X needs to be greater than four I'll store this and diamonds on the score length Now I'll have a glance of this view off diamonds length
14:38
So as you see over here if you check this length column which is X you see that the land first career than four millimeters So these were some basic data manipulation operations Now we'll go ahead then perform some detail visualization operations using the G plot to package So I loaded 30 deployed to package I'll type in library off cheesy plot to over here Now this data said BC which I'm booking with their diamonds Yes it is also part of the DJ plot to package
15:06
So if you want to implement or those same court in your system then you have to load the cheesy plot to package first Now I want to understand their distribution off this price column So I understand the distribution of this price column I'll just go ahead and build up his telegram So this is how we can build a history Grandma I'll type in Jeezy Blood and I will assign
15:27
though diamonds data said onto the detail you So once the Data Lear is done I'll go to the Easter Ticks Lear and I will map the price column or into the ex aesthetic Now I am done with the detail You're so I've mapped the diamond service it onto the date earlier and I've also mapped the price Call him onto The Exorcist addict Now it's time to determine the geometry
15:49
Geometry is basically the type off Lord which I want to create on Duh I would create a history Graham So you create a history Graham We've got the jump hissed a gram function Now I'll just hit Enter and we've got this really beautiful plot over here So let me zoom this right So as you see there are a large set of diamonds so they are around more than 15,000
16:12
diamonds whose price would be less than a round $1000 Right And there are very very few diamonds so there would be no living around 20 or 30 or diamonds Who's prizes girl in $15,000 So these were some of those simple inferences which we were able to drop by building this test a gram So now that we're done with the basic dealer manipulation on data visualization will head on to the main part where we'll be building a linear regression algorithm Now we want to understand
16:40
How does the price off the diamond very with respect to the carrot off the diamond So let's see if I given a carrot value off around 1.3 So the carrot values 1.3 Then I'd want to know the exact price off the diamond This is where Lino Digression comes in Now before we build a linear regression algorithm we actually have to divide our data set into training on the testing said So we'll be building our model on top of the train set and will be evaluating the results On top of the test
17:12
set Soto split the data set into training and testing sets I would need the CIA to Will's package so those CIA tools packets gives me the sample dot split function So let me just type in sample dot split over here and the stakes in two perimeters So the force perimeter is basically the column with respect to which I want to split my data set on Guy want to split this data set with respect to the price column so I'll select this price got him over here and I will give in the
17:41
split ratio So I'll set the split re sure to be equal to 0.75 And I will store this in a new object and I will name their object to be split
17:52
bag Now I'll just read the list commanders executed You guys will be wondering Why do you have to divide a data set into train on Dhe Best set So let's take this example to understand it better So let's say you have ah math test tomorrow and you haven't learned anything at all But your friend luckily steals the question People from the principal's room Onda When you see the question people you see that data fight questions and you perfectly learn all of those fight questions
18:20
Now there are two situations If all of those five questions would appear in the exam than your east the exams you'll get five out of five bite What if none of the questions appeared in the example You'll get zero or a fire Right So this is what I'm trying to tell you So we'll have to divide the data set into training Investing said should have to train your model on one particular set and you'd have to test it on another set on If you don't do that and directly go ahead and
18:48
build your model on just one particular data set Then there are very good chances that it might feel on a new data set So this is the reason why we divide into two particular data sets So here we've got this play attack Now let me use the head function and see What is that in the split attack So split underscore attack Now as you see you just got a bunch off through values over here So what dysfunction basically does is it assigns 75% off the records with the true values and 25%
19:18
off the records with the false values Now using this true and false values will divide this entire data set in tow Do parts So for this I'd have to use the subsidy function so I'll type in substance over here and the first parameter would be the name of the details said And then I use this plate tag I'll just type and split tag over here And wherever the value of split tag us equal do true I will store this in a new data set and name that data said to be equal
19:47
to dream Similarly I'll use those subset function again and I'll pass in the first parameter which is the name of the data for him On this time I will set those split back to be equal to falls So wherever the split tag is equal to false I take all of those records and I'll store the window That said Now we've gotta train set and the That's it Really Let me have a glance of the number of fruits in the train set and the test it So I'll type in Andrew Off Train and Andrew off best So we see that
20:19
there are 42,126 entries in the train set and there are 11,814 and prison does best said So now it's time to go ahead and build the morel on top of the train set and to build a linear regression model We've got this alum function so I'll just type in Elem on the first barometer is the formula where we given the dependent variable and the independent variable so my dependent variable as price After that I'll given though they'll assemble And then I'll given the dependent
20:50
variable which is carried So I want to understand How does price very with respect to the carrot So whatever value is given on the left side of the tele symbol that will be taken as a dependent variable and whatever values are given on the right or the tail assemble all of them would be taken as the independent variables Who is my first perimeter which is the formula And then I'd have to give in their data set on which I'd have to bring the model So I only told you we are supposed
21:17
to build this mortal on top of the train set and I will store this in a new object and I'll name it us Let's see Linear model one Let me just print it out over here Lean your model one Right So this is what we have over here So call You see that we have the formula price with respect to carrot and it is built on top of the tree inside and beneath that we've got the Y intercept and the M value for this independent variable So ex discarded and the M value or the coefficient off this car it
21:53
has 7894 and the white Indus it So why intercept for the question A sick will do be ex would be the e Val you were here So we've got the white intercept and we've also got the M value or here Now that we've built the model let's go ahead and predict the values on top of the test said So I'll type in predict now the stakes in two perimeters So the force perimeter would be the model which you've built and the second perimeter would be the data set on which were supposed
22:20
to pick the values And I want to predict the values on top of the that's it And I will store this and let's say my result So we've got the actual values and we've also got the director values Now I'll go ahead and combine these two So combine these two I'll use the C bi and function so the actual values are present in the price column off this test data set So I'll just name the Scala Mama's actually And then I will create another column which is predicted on this Richter Values
22:54
are coming from my result and I will store this and final data Now I'll have a glance of this So view our final leader So we've got the actual values off the diamond and we've got the predicted values of the diamond Great So let's see if this is the actual value 3 51 depicted us 54 So there's a huge error in this prediction over here Now we'll go ahead and find out the residual or the error in prediction so that would be basically actual minus predicted That's how we find the editor
23:28
in prediction Noah I'd have to convert this into a data stream first so this is actually a Metrix so I use as dot data for him and I'll pass final data into this and I will store this back to final later on Once we are done with this I will go ahead and subtract the pretty good values from the actual values So final data dollars actually minus final data Dollar predicted So I got this out I'll be justice inside these round braces and I'll stow this in a new object and I'll
24:05
name the object to be equal to zero
24:08
And I'll bind this era back to the seemed He does it So see bind off final data and I am binding it with error And I am storing it back to the same data set which is final data it So we've got the actual values They've got the projected values off the price off the diamond and then we've got the error in production So these are the individual era Sid deals with respect to each point Now we want to find out the residual some off squares so we have something whereas the
24:39
root means square error which basically gives us the average error for the entire model So as it is stated in the name fruit means squared error So first we have to find out this crowd off all of these errors so I'll type in final Data Dollar editor Let me get this out and I will go ahead and find out those square off this No once this is done So I have got the square off all of these editors Then I'd have to find out the mean off it great And finally I
25:15
will go ahead and also calculate the square root offered I'll pieced it inside this and I'll use the sq Artie function Right So rude mean square off error This is what I'm finding out And I get a root mean square value off 1002 121 which is actually quite huge So this tells is that the model which we built is not actually accurate for us to give us the right results And I'll store this and let's see r m a C one So this was a simple linear model where prices are dependent
25:52
variable and carrot was our independent variable Now we'll go ahead and build a multiple e new regression model so this time we'll use multiple columns as our independent variables So let me go to this data set and let me use the LM function again So price again would be my dependent variable And my independent variables would be Let's see X y and Zed which are basically the length width and the depth of the diamond So I'll type in X place Why bless Said Andi
26:22
I'm building this morning lugging on top of the train set so data would be trained
26:27
and I will store the Seine linear model tow I have built my model and I go ahead and click the value so I'll use the credit function on the first perimeter would be the model which I have built which is basically linear too And then I will give him though data on which I'd have to break the values which is new data So I want to protect the values on top of the test said and I will store this and my results too I have also predicted the values Now it's time to bind again the actual values and our
27:02
predicted values So the actual values are present and test dollar price and the predicted values are present and the my result too right now let me get them their proper names So I'll give an actual over here and I'll also given predicted over here And I will store this in final data too So we have our new final date already view our final data too It's so for the second model which you've built where we had three independent valuables These are the actual values
27:41
of the diamonds and these other predicted values of the diamonds Now let's go ahead and find out the error in prediction So before that we'd have to again convert this into a data for him So as dot data stream off final data to dollar process has to be final data to and I will store it back to final data too All right now it's time to find error So final data to dollar actual minus final data to dollar predicted I got this out and I store this in Edo Do you Now it's time to bind the era back to
28:23
the original data set So see bind off final data to and I am finding it with two which is stored again and final data So we've got the actual values though predicted values and the other in prediction Now it's time to find out the root Mean square a row for the second model So final data $2 Addo too I got this out We sit inside this Let me take the square of the rope One side B c squared off the editor I'd have to take the mean off it And once the mean is also done I'd have to take
29:03
the square root off it so I'll type in sq r d over here Right So this time though fruit means quero comes to be around 1574 Andi I will store this and let's say R M s E two
29:18
now let me print out boot our embassy one andare embassy do over here right So for the first model the root means where terrorists one due to one And for the second model the root means square address 1574
29:31
So for the second So for the second mortal the root means square error is higher This means that the first model is considerably better than the second model So this is how we can build multiple models and compare that accuracy with each other So we have successfully implemented the linear regression algorithm in our language Now it's time to implement the semen Bison and first full Bush will be walking with No ideas do does it So my first us would be to load the Bond
29:58
US Library so I'll type in import Fonda's speedy Now I'll be using the Reed underscore CSB function so I'll type in pt dot read underscore CS Week And inside this I'll give in the name of the file which is basically iris dot CS week and I will store this in a new object now I leaned it Object us Iris Now I'll have a glance of the head So iris dart head and the seleka middle first fire doors which are present in this d does it now This day does it is about
30:32
they're different species off the iris floor which are Sentosa wussy color on gua Jessica And these are the different features off these species So I've got the separate length separate battleland and petal word Now for this model I'd want to understand How does CEPAL length off the iris flower Very With respect to the battle island off the iris flowers so separate Lent would be my dependent Variable and petulance would be my independent variable So I'll go
30:59
ahead and divide this entire data set into their dependent variable and independent valuable So I will store the separate length into white So do that I've just given the name of the data from witches Iris and I'll give in double braces over here and I'll put in courts and I'll give him the name off the column which is separate lint So let me just type and CEPAL underscore length over here So I am extracting this separate length column and I am storing it in
31:29
tow The variable white Similarly I will extract this Battleland column from the Irish data set and I'll store it in tow X So I'll give in to braces over here and I will give in the name of the column that just battle underscore length and I am storing this into X So I have my dependent variable and independent variable ready Now it's time to divide thes two into their training set and the testing said so for that purpose I'd have to import the train test split matter from the s K loan
31:59
package So from esque a loan I thought it would be model selection so I'll type in Model underscores election import off train just split so I'll go ahead and import this matter from the S K loan package Now what has done I will use this function So a train tests plate I will pass in the X value and the Y value And then I given the test size so the best size would be equal to 0.3
32:32
So this means that 70% of the record score into the train set and the 20% of the record's going to though That's it And this would be slick It was four results So let me just type of those results So the first result would be extreme
32:47
And then we've got the X test off that they've got white green and then finally they've got white dust So what are these four variables so extreme as basically the training set off the independent feasible than X test is the best set off the independent variable Then we've got white rain which is the train set off the dependent variable And then we have the whiteness which is the best set of their dependent variable So these are the full results which are given by Dream
33:14
Best split So now that we have all of these it's time to Billy New Regression Morel on top of the train set So I will import a linear regression model from S Kaylen So I'll type in from esque alone dot lean your model I'd have to import linear regression So now that are important in your aggression I have to create an instance off it And I lean but incense to be our all right So I have successfully created an instance off this linear regression model and I'll go ahead than fed this
33:50
on top of the tree and said So L I don't fit And I am fighting this on top off extreme and white dream
34:02
Right So we have for this model now it's time to break the values on top of that test said so they've caught the credit function So l a dart predict and we'll be predicting on top of the tests it and I will store this end Let's say why underscore Fred Now we've got the actual values in the predicted value so the actual values are present in white pissed I'll type in whiteness dot head on Then I've got depicted values which are present in white bread but his own Mumbai curry So I
34:34
would have a glance of the 1st 5 courses to go from zero to five actually
34:40
Right So these are the actual values off the Zeppelin and these other predictor values off the separate lent to the actual value is 5.7 The value predicted by this model is 5.9 The actual value of 7.4 picked It was 6.1 that we've got 5.1 which was raped last 4.9 right and free Take this value Seven It was breaking us 6.15 So I've got the actual values and afflicted values Now it's time to find out the editor in production So I load the mean square error from the metrics office Kaylen So I'll
35:12
type in from escape alone dot matrix I would basically be importing though mean square So now that I import it I will go ahead and find out the means quite editor of this model And inside this I'll pass in the actual values which are present and white pissed And then the second barometer would be liberated values which are present in wipe red So that means quite letter comes out to be 0.19 which is actually quite less so It seems that this morning which you
35:42
build us quite good So in linear regression it's one of the most popular algorithms and production Because it can be used for classification
35:55
it can be used for regression It also happens to be one of the core algorithms
36:04
that is used by many of their algorithms Logistic regression is a linear regression based model
36:10
right If it takes support Vector machine another very famous algorithm and data signs It's a linear model Or if you take uh if it build do leaner models using been the data what it called bin B I and been data sets Then the modern that you get behaves like a decision tree right So many of the other algorithms It's very popular in the industry Internally they're linked to linear model ts You can spend your entire life's time studying linear models It has so much
36:47
of depth and breadth that it takes a lot of time to actually get used to all the six So today I'm going to start by introducing you to simply need models
36:59
the core concepts and then subsequently and leave some advance discussions for the next visit where we'll be covering F empty feature Injuring mortal tuning There will discuss some further at once Concepts only near movies only So I hope these people have loaded by My team has loaded the data set everything I'm going to give you three chords today Offiicial covered one in the class The other two is for you to go and explore further Okay Based on what is this This necklace This Peabody's
37:40
accessible to all of you Can you access this property I think she's going toe You can You can Okay The court is also available The people need the court of the data state You should have received three chords
37:58
All right let's start as the name suggests This algorithm is called linear regression on the reason why it's called linear regression has the names of justice It's based on the concept off a line a line a plane Ah hyper plane We call it by these names
38:18
So to begin with I need toe ask you whether you know how a Linus and presented line line in mathematics are you all comfortable with this All right so let's start
38:36
Imagine I give you two dimensions This is X excess here One off your independent attributes We call it columns in our day Different
38:48
right So loaded a data from our table into my data stream and one of the Collins in the data frame I collect X This is my independent variable The other column which I want to predict the target calling my call it away could be anything Why it could be the mpg on dhe X Could be the weight of the car The more the weight of the car the list of mpg Right So why is my different variables exist My independent video Imagine this X has 456 These are the units 12345 Think
39:31
these are the measurements off X unless a zoom for sake of discussion let's assume that when our excess one why is also one
39:46
So whenever exists one wise also one that is what you're seeing in the data You take another record there you see X is equal to then vice is old Vehicle is also equal to talk Ate the 3rd 1 excess three Why's is also three So when he tried to draw a line a relationship between them You see it's a straight line
40:15
The straight line tells you the relationship between X and why How are these two related to each other We in data science call these things as mortals Mortals are nothing Burke Sad faces in your feature space What is a feature space feature Space This collection off your independent attributes and the defendant attributes you're trying to explore How this why and this ex interact with each other That interaction is what be represented as models The mortals have
40:53
nothing but lines Surfaces hyper sir Faces in your future Space seven Evil kay Nearest neighbors Okay In Kenya's neighbors you have different classes of data points Suppose these are different classes
41:10
Okay on Okay let me make it slightly more Khan please What gay nearest neighbor does this We call it breaking your mathematical space into regionals It breaks a mathematical space into regions so it says this region belongs toa the circular point This region belongs to the Strangler point and yes things on the wrong side Getting nearest neighbors splits your mathematical space into regions Those regions are called Warner Regis
41:50
so this imaginary surface is the modern in Kenya Same linear regression This line which represent the relationship in X and y This line is my morning So I want to predict the value of why given the value of X what the line is saying is why is equal to X So the prediction ISS whenever excess some value Why will also be the same Can I write this expression as y called Juan IX Does it make any difference one into its this one In mathematics it has a meaning This one is nothing
42:36
Bert This angle the line makes with the X axis the time off that angle off that angle in tow X whenever why is equal to X This angle will always before different degrees Okay and in trigonometry ban off 45 years old This one I met him wondering What is this technology time beating on literally Jardins actually told comment Since look at this when we started in school became across trigonometry in trigonometry became a CRA signed Teeter costed out and Rita what are the signs
43:29
it'll cost You attended one of these things These are names given to certain ratios So Tank Tita is time means the ratio between
43:42
we show it as de vie over DX How much white changes whenever exchanges by 100
43:48
that issue's court homos Why changes when exchanges But one unit How much white changes in this stuff when exchanges by 100 so one by one is equal to one de vibe RDX How much white changes by small change in X one Entertainment eggs That ratio is called tan Tito Sounds very hi fi but that's what it is Okay so venom er
44:20
both of them synchronize whenever exchanges by one white in just buy one It will always happen when this angle is 45 So we say that Ana fortifies one So when you say 10 affordability Viber Dex will be won by one It'll always be one Golic let me modify this a bit So right now this expression is represented as m Okay This is a standard practice Very represented as am So why equal toe a mix iss the expression mathematical expression off this land Suppose I put a line here okay And it's parallel
45:08
to this line It's battle in tow This line Okay which means this line is also making 45 degrees toe the line pilot X axis I'm not going to show this link it okay Just to make it less cumbersome
45:27
So I have another line here This is L one This is L too Now can I see why critics When x zero Why It's not zero wise too correct When excess one Why is not one Why is three
45:50
when access to why is not too wise
45:54
for so we add a constant tow This
46:01
this constant is equal to two right now for this life This constant is nothing But what is the value off wife and x zero What is the value of why even x zero That s Corsi It also goes by the name off intercept Were Indio mathematical space on the white dimension They're on the my dimension Does the line intercept or cut the Y axis So this is the general equation Off line Waikato emits Lissy It tells you how why and excel related to them
46:53
If you look at this line L one
46:57
if you look at l one then this becomes zero So why you call it a mix If you look at L too l to this becomes not zero but two So this relation continues to remain missing In both the cases the Emmis seem em It's nothing but turn off the 44 degrees we call it slope Right So this is the Kendrick equation one way off representing a line y equals MX plus c So in this expression
47:45
why is given in your d Does it
47:47
It's a target column X is given in your data set It is the independent columns
47:55
they live in and attributes from this data that you have given to the algorithm The algorithm is found out for you the M and the sea We call these coefficients coefficients off the mortal this m and the sea
48:15
They reflect the relationship between Brian ex seniority does it so the M and the sea is what form steam ordeal for you Shall I complicate a bit more really complicated bit more supposed Instead off having one Or let me let me just continue with this family So you have one target variable boy one X In this case 10 or 45 happened to be one But suppose you're Linus like this It's a straight line but it's not At 45 degrees it's a 15 degrees Right
49:11
then in that case also the question off the line will be why Kal toh M
49:15
x The reason why I'm not putting CC zero here C zero you can put See what's easy You and this angle m This value of him will not be one It will be 11 living It's 44 degrees It's less than 44 degrees So Stan off 15 degrees into one into X plus c This will be a question of this life What is Stan Tita Tandy does nothing but how much white changes for one unit change in X We call it then Okay All right
50:02
Let's complicate this a bit more Suppose in stuff only one variable x No you have X one and you have another variable x two and you have only one dependent variable way In such case the algorithm will find out the relationship between X one and why X to end way So it'll express that relationship as why equal to m one x one plus m two x two and there will be only one constant Because now in stuff having a line you will have a plane near the plane go through the Y axis At some point it cuts the Y
50:50
axis at some point so now you have to adjust one edges on X to the other Edge is on X one The two edges may have different slopes Different angles with X axis Those are reflected over here is M one and the sea continues to be the same scene Where is the plane cutting the Y axis Which means what is the value of why when x one and X tours hero that is called interested Wait you have zero Also will be called a hyper plane We don't know how it looks but it'll be 11 Yeah so human minds can
51:46
imagine when the three dimensions but supposed to go beyond three dimensions They're also the linear models work Those planes are called hyper planes Hyper planes How do they look we can't imagine so we don't know that But they will also be one single plane and there'll be no ups and downs No curse Nothing will be street plane What do you mean by straight Plain in four dimensions Lawyer It led the same properties as the properties of a plane in three dimensions These excellent actually
52:24
be in a different one right 90 degrees this angle in Vina models this is a three dimensional isometric view This angle is always 90 degrees We say that there are struggling to each other Look at this laptop Okay In this laptop now this is my ex one dimension This is next to dimension There is a plane This plane is your model
53:10
This plane is making some angle I can't make it right now I can Oh right It'll make some angle with excellent access Will make some angle with X to access This is named to this is my M one
53:23
So at what angle The plane is two extra Next one will design the value of y Okay In this case the value of seyval ba zero Because the plane is starting the why access Why access is perpendicular like this It's touching the way Access at the origin The zero point some imagine I can remove this If this also you got Adelaide I could remove this Okay And imagine I'm putting it in some angle cutting the Y axis at some point Then this angle this angle is my name on them too The
53:59
point at which the plane cuts the Y axis must see We cannot imagine beyond three dimensions how the planes will look But they will have the same mathematical properties as a plain introduction a point that you should probably keep in mind It will become useful to you down the line when you're doing be planning since afoot the number of dimensions off model This is the model right now The sort of face the number of dimensions How many dimensions is this mortal dimensions lent in
54:36
the breath The model is merely to sanctions The feature space is three damage so the number of dimensions in a modern will always be one less than the number of dimensions in the future space Right All right Um
54:56
you can call it anything like that's not a problem Exp I see Okay That's what I say You want to call it Xanadu colleagues It doesn't just replaces Literally Basically these are independent variables This is a target So we're trying to understand how the target variable is influenced by these You have to feed off So how is the model dimension No extra neck Strict 33 independent variables So they have one target variables So that makes it for four By vengeance Morton will be three
55:41
dimension in darkness on however it looks I don't know what will be left 33 in just we don't know how to hold you So that's why we don't talk about it But the property's off That mortal will be similar to properties of a plane Alliance on For that is what the algorithms resume All the algorithms are zero that you're independent variables Okay Suppose you have a date A free In the data frame were multiple columns These are independent variable See everyone I ve to
56:21
every three and this is my target All algorithms assume that independent variables are independent off each other They don't influence easily The only influence the target that's what they resume But in real world that is not true Suppose I won't predict mpg off a car given the weight of the car Given the horse port of the car we know that they're related so we can build a model But we also know that horse poverty and weight are also related to each other More the
56:54
weight of the car more the hospital They're positively really But the algorithm assumes that they're not related this problem in linear models It's called the problem off Colin Garrity where assumption is each dimension is independent off the other
57:12
But in reality they influence each other That problem is called Colin Charity Okay this calling Garrity We're independent dimensions interact with each other They're not independent This Colin Charity can lead you to a problem when you production eyes your models The models may be less effective less vertical predictive power then required Toby because of the culinary So one of the things that we do when building models iss we see what the call in Garrity's between the
57:49
dimensions If the cooling Garrity is very strong
57:53
they're very strongly related to each other The independent variables Then we may have to resort to some techniques suppose this is my horse Pull over on This is the weight of the car on def A plot A graft between these two Can you tell me how they look This is weight of the car This is hard sport of the car If applauded graft between these two take all the data points You'll see some linear trend Okay All if you know what is our value
58:30
Confession of correlation Our value started Six You don't start a six Uh are really of measurement off how these two variables relate to each other How strongly you will see that our value comes very close to boss to one In this case the maximum it can be as one So we're very close to post it Okay Which means these two A levels are not truly independent off each other
59:04
But the algorithm wants all the dimensions to be independent a feasible So in such cases we resort to a technique called dimensionality reduction
59:15
which is not simply dropping the dimension I'm a convert them into a composite dimension Both the dimensions put together I create a synthetic dimension And I used that synthetic dimension in my model building rather than these two In this case the horse barn on the way is the same Almost It's invisible way if that removal is an option to you when the relationship absolutely line but you will see that it's generally not a line it's a spread on The spread is also covered So
1:00:09
the train should I drop Should I drop this or this That depends on which dimension it's more prone to It arrives when you calculate given me Katherine the later sense of being all that analysis I'm a convert into a synthetic dimension using a technique called PC principal component analysis Oh no no That's what I'm saying that it's our brain Our brain can see the little angels but the algorithm can morning working 300 images All the 300 dimensions mathematically will be orthogonal to each other
1:01:02
You please keep this cushion part for a minute I want to cover some ground before I answer that question because that's a very important question you're asked All right let's move on Oh the term regression indicates that we're going to predict some values Whenever we build models to predict values we use the word regression the term linear know you understand where it comes from So your linear model is nothing but a combination addition Off waited variables
1:01:34
this waited variable this another waited variable You're adding the two to protect the value fly It's a linear combination off one or more variables with weights The weights are called coefficients
1:01:47
Okay so once again that it beat What is this M one m bonus How much white in just whenever expunging just by one unit Keeping X to constant What is his aim to em To us How much white changes whenever X two changes were 100 keeping X one constant Okay so this is the relationship between X one and why this is the relationship with in extra when boy the relationship is how much whiting this whenever input changes But one of it that is a relationship and that relationship is what we call
1:02:27
linear models we already see in this guy in the morn So before we start building our models before we start building our models we actually need to analyze the data set on what I mean But this is suppose this is my date Is it I'll go in analyze Between this I've even on target ivy too and target Ivy Tree and target ricin off These is strongly related with the target It's supposed ivy too On the target
1:03:08
the distribution is scattered all over the place
1:03:14
It's like a cloud The data points spread like a cloud There's no trained You have seen the spread then In that case I ve to It's not a good predictor off the target How can you say that Take any value off Ivy too Draw a straight line for that value If ivy to your multiple values off target Which one should I take I don't know which one to take
1:03:44
So we say that I've even I veto in Ivy too Sorry We say that ivy to independent variable too It's not a good predictor off the target On the other hand
1:03:57
if suppose between say I V one on the target we see the data data points scattered in some form for trend Then we say that I leave One is a good predictor off the target So the first filter that you'll play in your data set is off All the columns given to you which calls strongly relate to the target Those columns which don't relate strongly with the target You're going to drop them from your analysis You're not going to build your linear model Okay
1:04:39
so the relationship could be posted or the relationship could be negative It could be downwards Also it I want a strong relationship Doesn't matter It's this meal exactly I don't give
1:04:49
but I don't want to include any columns in my model which have this kind of a spread What's is the way Okay leader mortals have built one leaving you see linear relationships If you see nonlinear relationships then you might want to resort to non linear models
1:05:21
Okay leaner models They expect linear relationship between Why on the independent variables then your model's predictive power will be very hard However as you'll see now those are all very perfect conditions in the real world We hardly have such perfect conditions right So sometimes even when the relationship is not linear we go up building the normal we'll see them don't you
1:05:50
Okay how do you measure the strength of the relationship between independent variable and the target For that we use a metric called coefficient
1:05:58
off Correlation How these two variables The target on the independent variable how they call relate how they relate to each other which we express using a character called our I'll tell you what this art is This our metric can have a value of plus one minus 12 plus one In this range the least it can have its minus one The backs it can have a value It's plus one
1:06:33
it cannot be beyond the strange Arte menu can be Zito Canvas Ito When l a spread like this your R value will be very close to zero When you're a spread like this you are really will be very close to minus one When you have a spread in the other way you are very very close to plus one Okay so I'll use an indicator off how strong the relationships are So let's understand this article you're slightly more be too the formula for our values This this is a formula for our value
1:07:23
Okay so shall we begin Don't worry about the formula known wherever the formula understand the concept All the formulas are there in the library We just need to call those functions and getting things done But get the concept in please Yeah our values A measurement based on the spread off the needle No Have you heard of scaling Scaling linear model is one more linear regression where scaling has no impact Whether a scale that it are not
1:08:10
it's going to give the same result Well if you know the formula for variance What is the form lovelies It's is I minus x bar You can call it Mu the whole square And okay I use it'll do one toe and let's keep it end But as always your views and minus one If sample sizes lists using minus one Okay all If you're okay with this don't go there with distance What is this variance That's what you need to understand that the former Thank you I don't spend What is it telling
1:09:01
you Always intriguing We're What is Central Valley What is X bottle What It's me What is the central value Don't give the formula Don't give in the formal I another formula
1:09:28
Central value is a value which represents all the data points that you have Okay So if I take all your heights addict up divided by a number of students in the class That'll be average height off a student
1:09:41
in this class Now How good this average is First of all why did we take averages Why do we need mean and media number one You do all those things that many that is started Sickle answer Yes you're right But many times indeed A science will come across a situation where you have a function and the function takes us input Only one value to predict something Now which one of the values should I put their 30 students in the class Each one has a different height right I want city order some
1:10:13
kind of court for all the students in the class A great learning stamped court No What size should I tell the manufacturer I can't give different sizes and one unit off every size It's hard to give some size ritual represent all a few so that all if you can fit in comfortably right That is idea Okay Inside situations we need to be in for 21 value So we What we do is we find the expert or the mute But the cushion is How reliable is this expert Suppose the average height of
1:10:46
the student in this class comes to 5.5
1:10:49
Some of you might be six photos You might get off under duels That boss I'm six foot broken I'm fighting for a five on five So how reliable your central values is based on what you're going to do for their actions How reliable your central values is that reliability is given by the mission of variants It's a very intense too large
1:11:13
the central values not reliable Right So the variance gives you the reliability of the central values out Aladdin and single So the former love of Aidan since this right Just look at the new military Let's look at the new motor This is the case when you have only one variable x No I have two variables X and Y I want to see how they very together How they influence is they're very together That is called providence
1:11:48
which is given by excitement is ex bark
1:11:53
We don't square it into why I'm honest Viber So these final take some in his exports multiplied for violence labor That tells me how the data points in the feature space
1:12:07
they very together that is called Koreans Independent variables Independent variables Independent variables should not have evidence Target variable and independent variable should have strong co variance Otherwise you don't use those independent when you're building linear models The target variable the y on the independent variables Ivy's should have very strong co variance But within the independent variables the co variance should be zero That is an ideal situation Practically never
1:12:47
happens All right No Give it the school villians No look at the formula for our
1:12:56
okay on the top in the new minute of your cinco evidence on if you look at it carefully the units on the top and the units on the denominator are the same This is also examine its export This will take Simon's This is why I am in this way But this also vitamins labor It is square number There's an under route
1:13:18
This is also square Tim Burton and Roop
1:13:21
So both the numerator and denominator are goingto have seen units of measurement agreed So if I have a kilogram buy kilogram case than the unit's cancel each other out it becomes a unit less quantity it is a ratio Our value is the issue Okay Now when you have a diagram when you have a plot like this between two variables could be a different variable Stargate and independent could be anything When I have this kind of float you will see that our values close to
1:13:58
zero We'll see why this happens when I have this kind of applaud various here trend But as X increases why decreases They're going in different directions X increases when decreases their opposing each other Then we love are really close to minus one whenever they go hide in hand When X increases Why also increases
1:14:23
They go in the same direction together Then R will be close to plus one on It's not very difficult Understand Why Let's look at this particular case closely I'm gonna look at this particular place to look closely Yeah this is my mathematical space or features piece
1:14:47
Both X and Y have some expert and y bar the average night expert away But keep in mind all linear models that will build linear regression models They will always go through the point where expert in the UAE but meat So the leader Morgan that we so a few minutes ago this model will always go through wherever they export And the vibe our meat it will be hints hints on to that point not going to like this fulcrum All right this one particular point
1:15:28
that linear regression models will always go through x bar Viber I'm really surprised and I find it very difficult to understand why this is not toward now Look at the look at the points over here Okay let's look at these points carefully Take any point late Let's take this point for this point What ISS excitement is Expert excitement is exported this distance This is my expert The vertical line Excitement is expert Is this distant agree all If you agree what is why I minus
1:16:03
y But for this point it is a distance between this point and the Y bar
1:16:08
I agree Excitement is export is positive Why Amanda's y bodice postures of any Multiply these two It's like finding the area off a rectangle lend into breath So excitement is exporting The Wyman is by far for this point is nothing but the area it covers Over here This is the area This area is a Boston media All of you Okay let's take another point at the bottom of the screen Said this one For this point excitement is expert Expert is greater than excite So
1:16:44
excitement six bodies Negative So for this point
1:16:51
excitement is eggs bodies negative Why am in his wife but is also negative Negative into negative becomes pasta So all the areas over here will be posted All the areas over here also will be positive I agree Take this point Excitement is export is negative because excite is less than expert negative Why am in his way But it's Boston It's a positive into a negative Really Given negative all these data points will have negative areas All these data points will have negative areas when
1:17:32
a total this upset by you See Sigma over here Signalman sum up all areas minutes sum up all the areas here All the positives on the negatives will cancel each other out The reason why they will cancel you to each other Lord it Each quadrant has same density of these points The points are distributed like a symmetrical circle So whenever our values close to zero That is because your new military is becoming zero new military Becoming zero means your distributions like this for a
1:18:07
given value of X I have multiple values off Why Which Why should I take I don't know So we say that X is not a good predictor off way On the other hand if you take another this plot if it becomes tighter then postive areas is greater than the negative areas So are you really will be close to plus one will become 11 living on the data points lie on a straight line which will never happen
1:18:41
If this plot was this way all the negative areas will overwhelm the positive areas that are valuable Close to minus one Right all of you Okay so this is a concept off our value So what we doing Lina Model says we test the our values off every independent dimension with the target variable We sell it when he does whose art value is either close to minus or not plus one Any dimension With our radio close to zero that dimension is a useless danger How close to zero is useless That depends on
1:19:19
your particular problem statement Incidentally any art of allure greater than 7.7 minus four Plus it's considered as a good very any variable with our value between 0.5 on 0.7 It requires for the research whether this relationship Israel Or is it a fluke by chance that you're seeing in your day Does it any posters here A long one It could be something in the way You selected your samples from the population that leads to the Not so the sensuous discussion Let's just isn't so for
1:20:12
our value to be zero Okay if this is my ex and this is not why this is a expert This is Y bar Okay Expert Vibert I'm not showing you the court in it for our very Toby zero
1:20:26
The data point supposed to give you 10 data points The data points have to be absolutely symmetric symmetrically distributed for our failure to be zero absolute symmetrical distribution in all the quarters What is it Probably that when you're doing random sampling you'll get data points in such a way that they get symmetrically distributed on this What is it Probably Brody Zito
1:20:54
That particular configuration of data points that you want from the universe but symmetrically distributed itself around these two experts and my bars to give you the arrival of zero that probably zero It'll never happen So even in the real world if there is no relationship in an expert in my bar because of your sampling sampling brings in some asymmetry in the distribution There'll be some asymmetry in the distribution You'll end up with an R value greater
1:21:20
than zero This are really recall it started sickle fluke So when our moon is very close to zeros it could be by chance
1:21:32
sampling problem It may not be a real relationship So you have to go in for that investigate whether the relationship is Avi Um it might be you How does it feel
1:21:45
if possible that will happen when you're sampling is completely gone case you're done some mistake and simply when then it can happen All right All right Shell No one I think I know how it is Yeah both cases we have to use this We have to use for both the situations just like the independent dimensions and within the independent dimensions knock over those which are very strongly correlated is the most only maybe this moment is your mole steals Okay Now this algorithm that we're
1:22:58
going to use what this algorithm will do is given the data set you have Okay so those this is the distribution x and Y In your data set you have only one independent call on the one target and you see notice the distribution is like this So this is a good predictor off this Why But let me understand In precise terms what is the relationship with my ex I see that there is a relationship with what exactly is that relationship precisely I want that
1:23:26
that precise relationship between Ryan and X will be our model represented by online in this case Okay The algorithm want this algorithm all doors it can find out multiple lines here Multiple lines can be is shown here Right
1:23:43
Want algorithm Louis we'll start with some random M and random See why you could tell him Explicit e Why you call them explicit is the expression You guys I'm talking to you Not that the Why is it that I am explicit E So by changing m and save an hour I change em and see the relationship between X and Y changes
1:24:06
which means the line changes the line represents our relationship So whenever I change MNC the line will change Okay this algorithm will start from some random M and random see and it will evaluate the different possible lines before it finds what is called the best fit line Okay Know what is best for Klein
1:24:34
amongst all the lines possible that it has checked best fit line is that which goes through the maximum number of data points it can go through all of them It goes from Maxim Number of data points on at the same time it minimizes the distance between other points on the line the distance between other points on this line this distance Do you see the distance over here This distance is called later What it means is forgiven value of X Your line says the value of why should be this But your data
1:25:17
available in your data set is saying the values This the blue line here is the predicted values for the different ex Actually values are the black dots So anyway have black dog about the line or below the line
1:25:33
That means there's a difference in prediction And actually the difference is called Okay don't mistake the word error Error doesn't mean anything negative but it's used for something that you already know I'll tell you wanted us so amongst all the different possible lines Little bit only one line which minimizes the some off all these errors If I sum up all these errors across all the data points the minimum some air of it will be given to you by only one line That line is
1:26:09
called the best fit Like the algorithm will find out for you from infinite number of possibilities The best fit line for you It's like looking for a needle in the haystack
1:26:22
And to do this it makes use off process which is called the greedy int distant
1:26:30
All algorithms use under the hood a learning process a process which the makings off to find the best morning for you in the given detested That process in this case is called great innocent What is it you don't want to that that constant classification This is a big mission How are the same concept will come into play here also in a different way I'll tell you what that way is Yes So we'll see that we So again once again in our apartment we're too far away from those points Right Come along
1:27:34
Any questions Oh yeah So the number of points is rarely know their differences more than than a pine tree This incident it'll take action I want you to go through a lot for the distance Most charity because there would be this Uh yeah it only release under points it on the language So it has to be taken together both of my equal priority But there's only one line which is satisfy that condition The driving condition ist there should be minimized That is the only
1:28:09
condition Okay so there are no two step process if it's only one step So let me explain this to you If I brought okay The question of Linus Weikel do em explicit See I'm going to take one simple case An explosive there in your original mathematical spacing out ex You have way One independent variable One different Incredible Okay so now we have two perimeters that we can modify And whenever I modify m and C I'll get different lines This is someone and see one This is M 22 this entry and see
1:28:48
three other different lines Each line has its own MNC Thank you but not all lines of good lines So if I have this line for example this is everyone If I take the little one l one is made off M one c one I see that many of the data points are away from everyone onboard the sights So the other even is very high If I take say the next one l two I like all this Anto I call this ELT Oh I see that l too has its own slope It's on Call this in Quicken missing intercept And
1:29:34
it also does a lot of feathers But not as much as even you do I tried various lines Then I come to L B If best fit Best freak line is this which has its own m best frit see best fit on E best fit on off all these mirrors This will be the least e best fit will be the least
1:30:05
Another question is by changing my m and C But just a small minute amount I'll get a different line So there are infinite lines possible but you'll have to evaluate Order the infinite lines here to find the best fit Brute force will not work here You can't use brute force See I'm actually time or anything All this gives its informant across the history I'm coming to that Okay So
1:30:48
in stuff looking at this mathematical space I can actually look at another mathematical which is related to this That mathematical spaces between M and C on this dimension is it Where other means some off all the rest put together Okay In the linear regression this terrorist calculated ass Why hat that its predictive value of oi minus white actuals Okay raise to power Too divided But an interesting number of data points Isaac we wantto end We call it square There some off squarely interests some
1:31:42
the sigma squaring dangerous So the air done by mortal is nothing but a sum of squared errors And this number of data points you have isn't this formula for villians So it it's nothing or comedians How data points very scattered across the best foot line The less of the variance more reliable the central valueless the lesser of the variance off the points across the mortal the better the moralists
1:32:13
Same concept comes to you in a differently
1:32:18
so some of square This is nothing with various variants off the data points across the morning So I want to find that mortal
1:32:28
where the variance of the data points across The model is the least Thank you Drilling music Cool And you say they have more radios for what it is Four dependent clauses
1:32:44
Insulation Okay that's wrong Great So the real cool But this can be a radius from
1:32:59
the neighbors or Dad Wait Which way India
1:33:20
She's thinking someone Everything you have is thinking which is nothing So if you're talking I don't I'll take your question on Just hold on a minute I'll take the question here
1:33:55
Written 1.90 extender division sent in this side The data points might still have noise in them which causes spread across the model you're talking about old class which is lying 1.9 600 division greater So by removing that yes you're cleaning the data but within the 1.9 600 division still you have noise that noise will appears scattered around the mortals LaPierre's variants around the morning Wait this morning with actually no you have asked a very very important
1:34:41
question which I will just answer this because this is a very important question which will be helpful down the line Just hold on a minute Atlantis mission first Okay Now if I take any access any variable X it has a spread Okay This bread variance Particle villians variants There are two types of variances
1:35:05
One is called stochastic variance which means nothing but probably stick villians The other one is called deterministic variants which means we're rings That happens for reasons that I know I know why that ratings happens Okay that is called deterministic But wear rings also happens in your data set For reasons that I don't know
1:35:34
we call them noise Okay that is gonna nice All right No If I have a plot like verse Alina model on have distribution of the data points around verse Okay For a given big for a given value of X Now look at this Actual value Is this in your data state but the model is saying it is this similarly There's another actual value Which is this Why is this variant suckering I don't know I don't know why that radiance is occurring This is called I don't want to use the words right
1:36:12
now Abuse it down the line This variance that you see for a given value fix your multiple values of why But they're very close But still there is evidence in them
1:36:21
that were insists called stochastic noise Okay for giving the love eggs the fact that value of vice spread around its central bar This is deterministic We know that there's going to be this way All right so many address the out liars outline is something which is lying to stand two divisions away or 1.96 tank division of E You're addressing outline Okay but you're not addressing the random variance which exists in the data points written Leave 1.96 and two divisions Those will cost a scatter
1:36:57
of the data points across the morning Prediction This wood lane actually is a black data points between the actuals and line There's a gap That gap is my stochastic I'm no unexplained evidence I don't know why you temples
1:37:23
Yeah in that too little deterministic is I already know my Y bar has despaired and my access desperate This spread is on civilians
1:37:34
I already know It has the spirit I know it will be like this Only I have no reason for worrying about that What Entering a borders forgiven believe ex wire the multiple values away That's so that is what noise is it because of lack of expertise
1:37:56
Nothing through your experience No one on the right Okay maybe When way have I did together in TV Dad is creating this cart off people back when the when the different variables actually interact with teacher of the school in Garrity Nor did I said you can uh you can have trouble with this noise That can happen The noise may get cancelled Don't or the noise may get magnified way
1:38:47
appellation of me Like Like I don't know what is least that's what I'm going towards Okay so I have different combination off M and C So I moved away from my orders in a mathematical space into another mathematical space where the dimensions are the different slopes and different interstates Different lines Okay Each line gives us summit of the former love of this terrorist is some off school letters which is nothing that variance across dealing when every
1:39:16
other square terms This is called quadratic equation in mathematics all quadratic equations they have property The properties called They're convicts functions convicts functions means then applaud them when implored them against their independent variables MNC They will always acquire a bowl ship three dimensional bowl such functions which acquired this kind of a shape a bold shape in the three dimensions Such functions are called convicts functions they're guaranteed tohave One
1:39:53
absolute minimal parable extractions one absolute minimum So for some combination off MNC I'll get the least ever that combination off MNC which gives me the least error is my best fit like so the algorithm will start from some random MNC So maybe this is the random MNC This is my random in and this is Miranda Massie When I use this I get a line But the sum of squared errors across the line is very high So it's point here is one line in the original mathematical space sand or is a mathematical
1:40:32
space between X and Y If this all smell one Maybe this is the little one
1:40:38
It had some m and some see that MNC is this point This point represents this line This line at some sum of square letters the sum of square terraces This So you okay All right No kidding Uh let's more I know what this algorithm does is it starts from some random MNC and it uses a concept called grading descent descent As the name suggests we have to descend Okay Granted isn't algorithm which internally makes you soft What is called partial derivatives Partial derivatives And
1:41:40
this term partial literally is I'll tell you ordered this It's not rocket science It is d Viber Deeks It's exactly the vibe RDX What He does this for a small changing am in this direction or this direction for a small multi engine in this direction or this direction from the first m the random in for a small change in m in this direction bx how much the other drops or increases so many changes in this direction I'm going further up there increases divi increases
1:42:20
Whereas if I go in this direction I reduce my m My device decreases psych notes tone that I have to move in this direction but they're taken one of'em right now What's the so do The same thing would see if I increase the sea slightly here What happens to mayor of If I see slightly here What happens to mayor I'll go in that direction where my error decreases descent
1:42:50
So using this concept of partial verities which is nothing but the vibe Urbi ex applied on individual dimensions one at a time One act of time That's why it's called Passion
1:43:03
It finds out it moves from this line to the next line Tow the next line to the next line until it reaches this point At this point water happiness whichever direction you move in Beyond this there is only going to go up I better go this side or that side It is not going to come down for the so it realizes that it has reached the global minimum This is your mystery people right Our district I used to be so does not evaluate infinite lines It uses this grading descent concept on
1:43:52
finds out for the next best fit line Next next next next on reaches a bluebird
1:43:59
this but morning word You know if you
1:44:10
get stuck in linear regression the other function being quadratic it guarantees you will reach absolute minimum Okay But in the process off jumping
1:44:23
there is them something or learning step Okay
1:44:27
They're too far away from this soil Not I'll just tell you the concept which is used along with special directors If the learning step the amount of change that you do in divine digs If the learning step is too high you might jump and oxalate You're my junk The global minimal going the wrong side and keep oxygenating backward and forward in front looks so the learning step is adjusted as it keeps jumping down the learning steps become smaller and smaller Various ways of handling
1:44:58
this your intoxication one of the basis I say jump towards your objective The goal The steps become smaller and smaller This algorithm is scored Bold driver I'll go home which is part of your great innocent one variant of great interest when your little deep learning this partial derivatives will become very important for you to understand how the noodle it looks work how the back propagation off errors happened Those things and come into play Okay but there's still a lot of time before we reach
1:45:33
there Hopefully I'll spend some time on this one day And But you have areas of the curtains We just apparently better than this
1:45:49
Okay we'll talk about those things down the line Uh Shelly Moore all of you Okay your questions are still parked right The two questions What are those Two questions Wait Why should they be No Colin what happens in business school That was one of the questions that came up I remember that Uh good So my strategy Okay let's move on We'll discuss those things in detail right Okay So keep in mind we're looking at this Data points closely the distance between the actuals and
1:46:49
the line The line is the predicted values
1:46:52
for a given value of X The predictive value of why is the blue line the distance between the actual video shown of the black dot andi line That distance is shown by yellow dash lines This distance is a measure and off OK the objective This Scram off these editors All these other squared That's some She'll be minimized Okay It is not very difficult to conceptualize once again So if I am I leaner model drawn like this best fit lying the data points of scattered
1:47:27
around this You look at the data point from this angle Then you do that you'll see a spread This variance This is going some off square letters This is nothing but excitement is expert Ex body here is you predicted Lynch so formally remains thing That variance is called sum of squared errors That variance We have too many mice The minimal Davidians ISS the better your modernise All right so let me explain this to you because you're going to come across these terms down the
1:48:10
lights So let me explain this to a slightly more detail We're looking at this morning
1:48:17
on the distribution of the points the rate points is given to you The blue lines are exporting Viber on Keep in mind your model The best fit line will always go through the point We're exporting by about meat
1:48:31
All this Okay well look at this Don't worry about these terms It's all I mean not very difficult Oh conceptually it's official is Yeah No If I take the red point red point is actual point In my data set in your data said in your data frame there is a record with value of X and it's got this morning value Why Israel But the one but the model The best fit line say's the value should be alone Be too the distance between B two and B one This is what we call her the variance This is what we call
1:49:20
some of scholars Okay I'm sure you only one data point here But you have to imagine there are multiple data points here Sum of squared error place all together Okay so this contributes to some off square There's another term that we use This is called Total later Total letter across all the data points to take the total letter is nothing barred The difference between vibe Are they expected Value off Why On the actual value for you
1:49:54
in your data there's a district distance between the actual value of why and the predicted value of the expected value of y expected earliest Wiberg That distance is called total later
1:50:10
off the total it of your model has captured this much Your model is an immigration moral Hence the name given to this is the regression off the total aRer your mortal is predicted this much other off the total every new data set Your model is predicted this much the difference between predicted under labour
1:50:36
This distance is called regulation What it has not being able to explain is this This is called some offs creditors some off school Dylan has another term which is called residue Littles Residues are those things which we don't have a reason for We don't know why they call They're called residuals Some of screenwriters is also known as a residual errors So many buildings models especially me goto your deep learning part in all things will come across
1:51:08
these tunnels Photo later is the distance between your actual data point and expected value Okay so it's average height of this class is 5.5 The difference between your height and 5.5 is a total error And I have built a model to predict a height off a person in the class based on the characteristics of the person So your model predicts some value for you so the model predicts so Hi Toby 5.3 your actual height of six
1:51:45
six months 5.3 that is 0.7 0.7 Is the different distance between Boeing seven Is this distance your actual height Minus predicted Hide right
1:52:01
The distance between your height and the class average that is a total It'll okay so you'll come across three terms taught a letter sum of squares and regression Keep in mind that a condition aided and this is the latest These two together is a total later
1:52:23
And don't mistake the word error Error means variants It means millions off the total variance in the data set I told you one part of the variances cards by the deterministic
1:52:36
The other part of the variances calls by the stroke Castaic long deterministic Something non deterministic is your recipe later Some of spirits also
1:52:51
thing thing in this case I'm sure you leave one point But you have to imagine all the points together because this s s e s s sir there across all points put together What I'm saying is what contributes to asses are necessary That's what I'm sure
1:53:11
So you have to imagine all the points that that becomes too cluttered That's What contributes stresses are what contributes traces See that's what I'm showing you Yeah he scares me I told the little Miss this I want to minimize this unexplained there That is what my objectives that grading descent works in finding the best fit line for you Their best foot line is that line where this terrorist minimized
1:53:48
because I don't want to a model which is not able to tell me why I want a model Ish tells me why this is so I go to do the traditional way Mister I don't like it because of aliens It's because of the natural radiance in the process Okay when you're running a process many things come into play Factors come into play because of its values Change here under
1:54:22
around expected value That is okay with me But for a given Elif input there multiple values of why that I don't understand forgiving aloof in production We've only one value of why Why are there multiple values of way
1:54:36
that I don't want that part It's my unexplained it I want a Marty which minute unexplained interest minimized I don't want a model where unexplained that it is very high What is the use of the more it will not predict anything from Okay You think given Malia you have one So the dishes is
1:55:03
people Peter's predictive value But here multiple values off p once spread it on the line
1:55:12
That gap is unexplained That variance is what we called stochastic random billions You don't know why it happens So basically the idea is to minimize SNC that's all the bottom Linus minimus ecstasy So increase the Isis are assisted issue That is what our objective it's Yeah All right let's move on So before you build the model you have to evaluate each independent dimension and see what is the R value Our value coefficient of correlation comes into play to help you identify good predictors given
1:55:53
the target Okay Once a built immortal I want to see how good the model is
1:56:01
how the level the model lists for that views Another metric that metric is called core fission of determination Munition on this is represented as our square But before we jump to it I should caution you Our square is not always our into our r is a question of correlation so naturally to think that our squad will be out into our that is true one Lee Then you build a simple linear model between one damage in one target It's only true at that point in
1:56:50
multi dimensional multi variant analysis very have more than 23 dimensions You can't see our Squire's ardent o R Okay so ask what is just a symbol that represents another electrical shop Your determination how much off the total variance in your why has been explained by your model Keep in mind the total variance and why is consisting of two components deterministic non deterministic stochastic random How much of this total variance in your wife has been explained by
1:57:27
your model That measure is called Coalition of Determination obviously a model which captures explains maximal off Bavarians And why at least behind very small residue Lt's that model is the best one So are square It ranges between zero and one zero means your model is in order to explain anything absolutely useless One means you're immortal is able to completely explain all the variance in your target There's no residue
1:58:10
Such mortals don't exist We can never build such models Usually our valley will be between zero and one The closer they're toe won the bet of the modern lists because uh it's in simple ian models It's a square term It can never be negative But keep in mind in Multivariate analysis it's not just a square It's something more than that So long This range between zero and one Are you all OK How much How much Millions in white It's a this is eight That means your model is able to explain 80%
1:58:48
of the total radiance in your wife total idiots which includes stochastic on deterministic Right So you're Oscar should be as close as possible to one your model Very good So this metric is used to evaluate the performance of the mortal while our is a metric which is used to evaluate the independent dimensions building the model dramatically Okay if you take one simple linear model off one x and one way Okay That's what I'm showing here So this is where your expert and y bar
1:59:28
meet Please Look at this carefully This is my wife Barb This is my expert Okay So there's an actual data point Excitement is expert Is this Why I minus y But is this that gives us an area rectangle
1:59:44
We saw that early right That area represents the total brilliance in your data point The total variance in this data point the co variance in the data point because off X and boy this is that area The light gray area the light gray areas This it's behind this dark grey You get the light Grady is this That light gray area is a germ A trick representation off the total variance because off X And because of why in this data point off this total radiance
2:00:19
for a gin mill of ex immortalised Pretty thing of value of quite a bit This
2:00:25
so this predicted value Why minus y bought x x bar This dark area is the explained the Syrians How much of this total variance has been explained Captured by your model
2:00:44
Our square is the issue of the dark area Too late area what you're seeing on the talk This is the residents unexplained millions
2:00:55
So our Squire is again an issue We have their issues between the total variance in the data points on dhe The variance explained by your model that ratio is cold Obviously
2:01:10
the lesser the residuals unexplained variance the better the modernise Yes I explained to you the problem is go back to our discussion on our where I told you If there is no relationship within these to the target and they have been variable you have to give get a perfect symmetrical distribution But the problem is getting a perfect symmetrical dissipation is zero
2:01:51
which means you're going to get a symmetrical distribution the moment you have asymmetric distribution Your are a little bigger than zero magnitude this way or that way which means our square will be greater than zero are square It's not considered as a very good metric for evaluating the models because it is easily impacted by fluke relationship So we make use off another metric their school adjusted expert which is nothing but our square minus fluke There is a formula for this How do you find
2:02:36
out the flu content which I'm going to avoid right now because it's too detailed to discuss in case you're interested Let me know I can share it with you Conceptual is easy to understand but the argument is very lengthy We won't have time for bed That's why I'm going to ignore it right So the metric that will make use off to evaluate our models is adjusted Our square knot are square the beauty of a distant ask who it is They're never in your model
2:03:15
building exercise to predict your target when you include useless variables Variables which don't really have any significant impact on the target But their relationship is a statistical fluke When you use such useless variables our square will keep on going up and up and up It will become It'll come close and closer to one
2:03:38
but a district Oscar will go down in Greece So then you young include useless variables in your model and just for our security will go down It'll reduce whereas our square it's likely to go up and just as quickly go one leaving you include good variables in your models Good variables are variable Swiss relationship A significant strong The views are just on our Squire for evaluating our mornings One thing that you need to keep in mind constantly getting clear Any attributes that he used
2:04:18
in your model building any attribute that attributes will always have both to compliments Good are on fake Are it'll have both the conference
2:04:32
But in a good attributes The percentage of this will be very very high comfort of this in a poor attribute Useless attribute person does this and this will be almost similar
2:04:51
That's why when arguably is closed to say 0.5 point six part off it will be caused by fake our part off will be real are homeless Israel Hamas is fake I don't know That's why it calls for further investigation before you decide to use that in your model an artist close to point by 0.0.4 point three we don't know what how much of that is fake How much of that is real We don't know That's why you need to further investigate how explaining the marquis killing causing you some way when you
2:05:35
use random function to sample from population that
2:05:39
since they are discussion whenever you do sampling
2:05:42
any technique you use clustering layered sampling mixed sampling border your sample will never bear to representation of the real population will ever be
2:05:52
so It's distributions of recycler different from the distributions in the population that stands in distribution is what will introduce fake arse So live sampling which introduces Did I show you that video will assure your universe to be under flux Continuously changing You know it's never started Various Your sample is a snack short I'm afraid of you I didn't Okay I'm just taking you through The concepts are not telling you why What is the formula for all
2:06:33
the things They'll become too heavy for you right now But I believe in Chile Take you through in case you're interested Some exemptions that this linear regression models make We already discussed this assumption off legality The relationship between the target and the independent variables are expected to be lenient Oh that brings me to the point Please be careful when you're our value is close to zero What does it mean Okay I think that's always ended up
2:07:14
saying there is no Lena relationship with an ex envoy Okay when our values close to zero bitten by an ex that means there is no linear relationship That does not mean there is no relationship For example if I give you a distribution like this this is X And this is why And this is parabolic There is no Lini relationship When we find our value for this decision to come close to zero But they're related There is loneliness relationships So whenever you do this analysis
2:07:48
are released close to zero Go back and check your pair panels Airport is the relationship non vineyard If the relationship is nonlinear and you're building a linear model in the curia Mystic A fundamental mistake Okay sanely if you have a distribution like this are very real comes to zero But there is a relationship written by an ex a sine omega T
2:08:15
is that you do scientist You don't want to die a ce so the anomaly is close to zero That does not mean there is no relationship It means there is no linear relationship If you have such attributes in your day does it Then you may have the think twice before you see any importance
2:08:39
So So unless you got this pair will not come to know that we have this kind of modeling is so is it always a vegetable for 100% If you ask me along The same pair is the most important tool you have in your toolbox This you should use to understand your later I suppose they got something different models Maybe I only need modern Okay So land Now that brings me to a question What is leaning on what is known Meaning Okay now look at this This is slightly funny on at the
2:09:19
same time something that is actually serious This y in X Okay Mathematicians will see it Suppose this is X This is why And the expression is bicycle to X square Mathematicians will say this is a non linear function
2:09:43
Data scientists call this linear function Okay What We can replace this with some value Okay So why is it okay So I can convert this into a leaner and then be leader models on this So many bills Linear models Why Eagle toe M X plus c
2:10:02
The reason why they're calling in here mortals is these coefficients They're not raised to the power They're raised to the power one That's why they're called Lena their linear in terms of coefficients If I have a mortal like this y kal toh m x squared plus C for us It's leaner Morty Okay so don't get confused with this A lot of my revolutions This Notley neighbor for us for us indeed Assigned Psychic learn or are This is Selena Moore insures a drink You'll see You'll see that if you do in our
2:10:46
time pretty case in order Mpg case Impeaches the target on your excess sale weight of the car You'll see that the data points are scattered like this You can still use
2:11:27
nonlinear see matter Goto nonlinear mortals are mortals Were these coefficients themselves Are complex questions They're not simple question They're not simple ratios There's something like E race to pour I Tita and all this kind of things will come over here you have Wait so you're allowed to try both Build a linear and known we never see which general Ice is Finally what is the objective objectors it should have minimal by us millions Which gives it that
2:12:03
minimal bias 1,000,007 foot back com voice what we call on somebody techniques in production We never put one single model into play We all this put a multiple models grouped together in a play we always installed multiple Morgan's particular Our own Good morning No that's okay Cameras facing you know you It's okay All right Okay But don't knock the camera off He is also sleeping Uh right So what we do is have you seen this porn binary code or apathy Have you noticed that
2:12:44
the audience never go wrong Though the audience individually may not have very high I Q But put together they rarely go wrong This concept is called wisdom of the crowd The same concept is used in Gaeta Science Also in reproduction Isar models We don't put one single modeling to flee We always put a collection of horror Now just go on somebody
2:13:05
The algorithm assumes that very selenia relationship between independent variables on the target It also assumes that there is no relationship between the independent variables that all algorithms do There's nothing particular about this algorithm Assumption of normality off editor distribution This algorithm assumes that the model that you build the best front line the best front line would be such that the variance around the best fit line will have normal decision What do you mean by this If
2:13:42
this is my best foot line and the data points of scattered it on this and when you look at it from this angle
2:13:51
It'll ever normal decision This is not guaranteed
2:13:57
Sometimes you indifferent the best foot line on the best front line You might see distribution like this More errors on the positive than on the negative This happens when have out liars in your data Six This algorithm all algorithms will go down on their knees when it comes to our class But this particular algorithm is prone to outlasts if an Oakland somebody new data set the best fit line will get pulled towards that get pulled towards will gravitate towards out lives Okay so when
2:14:35
that happens the best week line that you get the m and see that you get that MNC may not be the right M and C for a production It's a model may perform Poor leader Okay so any buildup models linear models any morning the first thing that you do is handled outlast first Okay
2:15:01
The various ways of testing this out whether this is happening or not one of the ways of testing this also you do a scatter plot between the actual values off why and predicted values of way okay with an actual values off Why end predicted values off Why Since the data spread out on the best foot line there should be no correlation The distribution to be scattered like this predicted value of my actual relief right There should be no correlation between them when there's no
2:15:31
correlation But in actual and predictor documents of data is evenly distributed across this desperately Sometimes you might notice there's a trend in these errors There's a train to some kind of a train in the in the difference between actuals and protected their trains It means there is a trend in the residuals as the value of X increases the residues increasing magnitude
2:16:00
When you see such friends in the residue lt's that means you're there's something not correct with your data sets Maybe you're having outclass Maybe you're building models on non linear data sets Those kind of things Would you come out I don't know whether you heard this word homos Capacity off Others Almost capacity had close capacity There are two things that become a prison statistics What it means is as you build a model the others should not
2:16:35
spread out like a cool They spread off errors across this model for all the range that you have The spread should be more or less a uniform Sometimes you will see that as the value of X increases for the best foot line the magnitude of the other increases the spirit is still normal But the magnitude off terrorists residues for high relief X is different from the magnitude of errors for lower value fix They're different in variables X sex Things indicate hydro scholastic whom a
2:17:13
scholastic The expectation is it will be home musket as Homo means homogeneous Okay so the residents respective want a range of X is will be similar across all the range assumption off independents off it us I mean ever done in one recordist movie influenced by the other than another record A drill down on one dimension is no way influence but the other down another dimension The dimensions are independent of each other That exemption is what is usually violated but still regard and build our
2:17:49
models I found a very good water call site which discusses these errors I'm getting to the Ural here by any chance have done this Darren Morton pissed started sticks You don't statistics right The influential And uh this thing Descriptor started six inferential statistics Sudan's ease Course it is You're done All the fetus and all these things have done right Okay if you're done those things And if you remember those things maybe you would like to visit
2:18:21
the site A lot of information please expand our horizons Go beyond what we teach in the classics That is just not possible to teach everything in the class for the time limit So go beyond the whatever treaties Expanding Horizons Yeah Dr Smart will give you a very firm sound footing into data science and you'll be able to face situations confidently So wherever I come across any Urals any articles I'll share those things with you on this late advantages We're still not out of the
2:18:57
words There's still much more to come We're not It learned the full stuff but we'll do hands on and then we'll see the other stuff Advantageous is very simple to implement
2:19:09
and it's easier to interpret Provided the number of dimensions business So when I have a question like Michael M one x one plus in tow extra plus c I can give physical meeting tow this I can say for everyone Unit increasing Explain how much white in just that is given by M one cycle Physically interpret this I can give physical meaning to this question You can't do this in all the cases You may not be able to do this supposed areas Neural networks You
2:19:41
can't interpret those models You don't know what they are They're all black box How we can convert this whatever this formless telling me I can map it Jewel English right I can tell you in the day to day language but it is telling you I won't be able to do that If I use a random forest you're going to do ensemble Very well Do Under Forest known for this is a black box Morning We don't know what exactly you're doing inside support Vector machine is a black forest We don't know what
2:20:11
black forest Blackfoot is A pastry black box
2:20:16
We don't know what it's doing inside Right This model is a white books Outliers can bring it on to its knees This particular model is prone to outlines our linear models A pronto class decision tree is not prone don't lives And when this is very important point Since the linear model goes through the point We're expert on Viber Meet the best reclines The lines have alerted for you Go through a meadow eh Barton Viber meat If your expert in vibe are not reliable then
2:20:54
your bottle itself will not be re level
2:20:57
So if I built a model on our dimension Why And X If this is a very large spread many large spring The expert is not reliable An expert is not reliable How will the model which is going through the expert making use of this expert hold the model but alive right So the more time you spend in selecting your attributes to build models the higher the quality of the attributes that you used to build models the better your models will be and the last boundaries are Lena This is a problem not
2:21:33
in regression There's a problem in classifications Okay So if if you think obviously need model for classifications also So if you're having a distribution of classes like this okay and then this And you're trying to believe your classifier Sze you'll get probably a line like this as a best fixed line So I've always a strangled below This is circles and one trying list Whereas on the other hand if you have made use off nonlinear stuff long linear models then probably would have gotten a
2:22:08
better result Maybe you were guarding something like this Okay so if you're using only models it might go better classifications non linear models
2:22:23
So this limitation is not in regression a decent classifications Let's do one hands on Okay The Hansen is not auto MPD data set artists Acids didn't give in here It's the other one with a simplistic So we'll start with simpler ones the other one which are given to scar Okay Carpets A linear regression I've taken the data set directly Found the U C A Site It's called imports underscore imports hyphenated $5 data This is the name of the beautiful Okay so I'm still dogging
2:23:01
it I'm definitely reading from there so let me begin Mac blocked Liv is an instruction to the Jupiter notebook The plot Make all the plots in the notebook itself Sorry Jupiter's over I'm important Bond Us lump I on from psychical and linear model libraries I'm importing the linear regression You can have linear classifier is also Then I'm reading the CSB file which is available in this You are ill
2:23:34
It's a comma separated file on dhe I've taken all these column names from the usually does it So that gives me the data frame which I called Card underscored If there's no dutifully all of you okay but this is like loading a date and your table selects look att to records Transpose is not a must But since the number of columns is huge I put a transport I'm seeing 1st 2 records record zero and record one
2:24:11
There is obviously missing missing their time Some of the columns and their string data types in other columns Fuel type gas It's string that time Number of those two It's a string later time All right no problem Go down Our object is to predict the last column Given all these independent variables can we predict the price of the car Is there a relationship in price and all these variables That is what we want to find out
2:24:42
What is that The relations But before you do that look at the data types Many of the later times their object object means string machine learning algorithms cannot handle string data types They're to be converted to numbers So what I'm doing here is I'm gonna cover it in the numbers But before they convert numbers I'm doing I'm dropping some of these columns The reason I'm dropping these columns is if you go and take a frequency count on these columns Fuel type for example Our
2:25:18
engine location for example Most of the records have only one value for this This is called low variance filter Look at all the data provided to you in the data You have columns we're in the columns There is not variance at all There's a gender column Gender column All the data is about meals
2:25:39
then that Colin is useless It's not going to give you any information Such such columns are called low variance columns Though they're supposed to have multiple values they have only single values are few hours We remove them from the analysis So when I just did this
2:26:07
I wanted to this I've manly bro's the data Fine This is Excel sheet See if we find you can actually see let's do it nowhere We're talking Let's do this Um What did Louis go to the site on Let's download this date Isn't control See Okay
2:26:36
don't It's stupid Noble courts Oh thank you Now we'll save this down and save it as Ah well say Vegas Let's David is takes document and then we'll load it up
2:26:57
on this cold Okay But I'm not stealing it Unsafe to invest All Okay Done Let's open it Okay let's start Excel Excellence Running into problems It's evil Oh God Okay okay It's going to clash with their Ah recorder
2:27:25
I'm not going to open this Let's get on to this court But do this analysis you'll find that these columns have having low very inspectors They're fit to be kicked out
2:27:37
All right so then I wanna do is I saw the car data types The column later times Most of the data types that object string times I dropped my useless columns
2:27:53
Number of those Most of them are four doors then fuel type Most of them are giving the filter pass gas Injun location Most of the Miss Front one or two is in the rivers so on so forth So once I dropped them I'm still left with some object later type number of cylinders His object Andi any other off dictated I know only that is object types singing really really dropping Which means that the current price of the car is not a function of that Its function of other variables Okay look at
2:28:42
this Hey guys do you understand why I have a drop low variance filter The reason why I've dropped Lovett inspector is the price is not influenced by this This particular column which is love aliens If the price was impacted about this column Mrs Lowe evidence then all the prices should be fixed Right Which means the variance in the prices is because of the other columns not this cold So that's why you drop it now The number of cylinders What I'm doing here is I'm
2:29:13
converting all these into numbers So wherever I come across a strain called one I replace it in numerical Want to replace her two sons You should do this with care Okay
2:29:27
You should do this with care Suppose you had a column called Gender on The gender was male and female Then I cannot do this blindly like this because if I replace a gentle sea a male with zero and female with one that I'm introducing an order in the column that's called orginal columns Zero is lesson one but there's no such relationship in the real world So if you have a gender column then I will use a technique called one heart According we're not in
2:29:57
cording I'll use that technique to replace my strings into numbers That example I've given you an auto MPD data sit Okay you look at that later On the other hand if I have a column called Income Group where they have income groups like Low Middle High
2:30:16
Not that I can go and replace low income Group zero medium income one high income for two because the same order which exists but 01 to exist in that column Also
2:30:28
Okay so if you call him is an ordinary data type You can go and introduce order in your numerical values If the car colonised not or internal gender column then you cannot blindly convert them into one and two You have to resort to one hardcore in psychic learned there's a there's off facility function called label in quarter level and quarter introduces order in your later So be careful when you're using that in suffusing label in quarter Useful not quarters All right Way E i
2:31:17
e I just Yeah Oh so be careful when you converting your ob dictator type in the numbers Don't end up introducing any order which doesn't exist So when you use uh one hold according have you all have seen one According anytime you're not seeing or not according Okay I'll tell you what one according is Then we'll move on Sorry You have a column Gender on the gender column Has
2:32:11
meal on female Okay this I cannot use as it is in my modeling So what I do is I convert this into numerical data types All later types have been numerical
2:32:24
so that I can put in the numerical I use one According what happiness will replace this gender Colin with two columns gender underscored M gender underscore If Okay wait over the recorders and you'll have one here Zero here
2:32:46
Wherever the recorders female you'll have one here Zero So this column will be dropped in your latest It you'll be left with this call One and zero will be stored on the hardness or in the invariable restoring it as numbers Last class used that way Wait
2:33:41
wait He doesn't have one Borden still gets right No object So I'm dropping my these data types Okay On Dhere card data types Let's see again Borden stroke is to logic dictates Okay here I'm converting Must selling those into an American dead to rights on Dhere I'm converting my data types into uh this thing flute Okay guys all of you with me all the cylinders till cylinder point Okay After cylinder point What I've done here is based on the experience ahead in the other
2:34:32
Data said where they were missing values in autumn Pretty detested You'll see missing values Okay So what we do is if you come across any missing values replace it with man Nine means not a number None values Okay So what I'm doing here is guarding the school Dave don't replace Very become across with some question Mark Replace with man But I didn't find any question marks in this data set The question marks were there in order Imperially does it Okay so we don't We
2:35:04
can actually delete this line we don't need it Then at this point what I'm doing is they don't question box There we are Sorry Very questions They're correctly saw it here Right We saw it here Question Mark Say sorry Mr There are question marks here These question marks no would be replaced with not a number values Okay When you replace Cushion works with values you can do numerical python operations on it and converting the new appropriate values Okay that's why we did this move
2:35:39
on my data types right now are after changing the data types here after dropping the columns here some of the data types are still object I've just been string data types
2:35:53
So what do you do here Is I
2:35:57
first convert my cylinder into character into new miracles Then I changed my data types here Look at this card in the school Death bore I replaced that asked I float I replace Stroke column asked I float So I'm changing on the data types to float because many of these columns Let's see which column headed Kushner normalized losses normalised losses It's already a numerical calm I think Yes it is already a numerical column So I don't need to change Very go It's only new medical
2:36:34
columns I don't need to change it Rest of the column Sicher ob Dictator types I'm changing into floor types till this point Is it okay Once obtained the data dive through numbers Numerical I know there are very missing values So the strategy I'm using here for missing values is replace the missing values off Price column With the 1,000,000 off the price called I won't explain this stuff to you
2:37:08
So I have a have a daily thing
2:37:11
And in this data fame I've been called
2:37:15
This column is called Price in this calling their many Innis any any Wherever there are question marks we replaced all of them with enemies The question mark has been left behind by the data entry operator because when he was entering the data he did not have values for those columns Those records So every place question marks with any soldiers became a leader because no these have become any I can use the function filling me What dysfunction does this If you see that Tucker's in
2:37:52
tax What Dennis find out the median off this color What Our values are available using those values find out the median Replace the anus with a 1,000,000 off this car It is always safe for three please missing values with any sort of missing the lives with 1,000,000 because means they're easily impacted by out liars More this using classic categorical variables we're doing running with floor variables so we always replace mean or a 1,000,000 But we don't
2:38:30
have checked out lives All columns will have out liars mean get easily impact about lives
2:38:36
So if replace these values would mean ongoing
2:38:40
replace remove the old pliers than this palace will be completely incorrect Three plays with 1,000,000 millionaires Not impacted voters Right on the beauty off This is in psych It learn I confined median off even an object The column is supposed to be numeric but because it has question marks the data frame is converting your objectivity But you can still go out and find a 1,000,000 off the cold Why Because 1,000,000 is physical Central value mean is
2:39:14
mathematical average It's a calculated central value you cannot find mean often object data type You can find 1,000,000 often objectivity right all of forgetting the difference that is where the difference between fight on and our comes in are a lot of first converting numerical value even to find medium Now how are you How do you address missing values Okay how do you address missing values Depends on your analysis of the data and why the values are missing The values can be missing because of
2:39:56
some random reasons that we call it randomly missing values values which are random You don't see any patterns in them They're missing values in all records and all columns randomly Distributor
2:40:08
Then we say that data's missing at random
2:40:12
There's no particular reason for it Okay we're us There can be situations where the values are missing because off some reason they're not because of randomness For example if you take income usually will find nobody wants to disclose their income So I will find missing values in that income Call many off them So depending on what is the reason for the missing this you'll apply different strategies If it is income column where the values are missing
2:40:47
because of the nature of the column then I might have to build a sub model
2:40:53
a model inside of modern where I predict the Valley's off the income based on the other attributes available So my domain knowledge will come into play which will help me understand
2:41:04
what are the other attributes that this income column is going to rely on depend on How can I guess it So whenever I create such imputed values as we call them I'll create a new column income imputed So I will not touch the original column I'll create a new column called Income Imputed where all the data available is copied as it this including the missing values on the missing values are imputed from the other columns that is one started you If your data is
2:41:37
missing it Randall then replacing the column with a 1,000,000 is the default strategy views Look at the spread of the date on that column If you see long thing tales Venus Notre Level It's a very place with medium
2:41:55
right There is a package available called mice
2:42:01
It's available both in our and python in Python It is available in another package called Stax Morton Stats Mortal mice stands for multiple amputations True chained equations Multiple imputation sze through chained equations Yes when you run this of vertical package on your date is it It'll automatically try to replace the missing values in all the columns with predicted values were predictions will be based on the other columns will establish the relationship and see which one A
2:43:03
strong relationship going impure So one columns missing value might be imputed multiple times from multiple calls He's going to read about this It's a very powerful package and something comes with power You have to be very careful in using it So read about this mice I'm not gonna cover it here but it's a very building package You should read about this
2:43:27
and just to add to the confusion apparently the mice implementation in Python is slightly different from mice implementation in art No they both call it
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat