Skip to main content
Loading Explore guides…
PyTorch Deep Learning · Acedra
Home
Explore
Computer Science
PyTorch Deep Learning
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
3,120 segments
Copy
Hide video
0:00
This comprehensive course will teach you the foundations of machine learning and deep learning using PyTorch. PyTorch is a machine learning framework written in Python. You'll learn machine learning by writing PyTorch code. So when in doubt, run the provided code and experiment. Your teacher for this course is Daniel Bourke. Daniel is a machine learning engineer and popular course creator. So enjoy the course and don't watch the whole thing in one sitting.
0:28
Hello, welcome to the video. It's quite a big one. But if you've come here to learn machine learning and deep learning and PyTorch code, well, you're in the right place. Now, this video and tutorial is focused for beginners who have got about three to six months of Python coding experience. So we're going to cover a whole bunch of important machine learning concepts by writing PyTorch code. Now, if you get stuck, you can leave a comment below or post on the course
0:57
GitHub discussions page. And on GitHub is where you'll be able to find all the materials that we cover, as well as on learn pytorch.io. There's an online readable book version of this course there. But if you finish this video and you find that, hey, I would still like to learn more PyTorch. I mean, you can't really cover all the PyTorch in a day that video titles just apply on words of the length of video. That's an aside. There is five more chapters available at learn pytorch.io,
1:25
covering everything from transfer learning to model deployment to experiment tracking. And all the videos to go with those are available at zero to mastery.io. But that's enough for me. Having machine learning and I'll see you inside.
1:45
Hello, my name is Daniel and welcome to the deep learning with PyTorch course. Now, that was too good not to watch twice. Welcome to the deep learning with
2:01
cools at fire PyTorch course. So this is very exciting. Are you going to see that animation quite a bit because, I mean, it's fun and PyTorch's symbol is a flame because of torch. But let's get into it. So naturally, if you've come to this course, you might have already researched what is deep learning, but we're going to cover it quite briefly. And just in the sense of how much you need to know for this course, because we're going to be
2:26
more focused on, rather than just definitions, we're going to be focused on getting practical and seeing things happen. So if we define what machine learning is, because as we'll see in a second, deep learning is a subset of machine learning. Machine learning is turning things data, which can be almost anything, images, text, tables of numbers, video, audio files, almost anything can be classified as data into numbers. So computers love numbers,
2:55
and then finding patterns in those numbers. Now, how do we find those patterns? Well, the computer does this part specifically a machine learning algorithm or a deep learning algorithm of things that we're going to be building in this course. How? Code and math. Now, this course is code focused. I want to stress that before you get into it. We're focused on writing code. Now, behind the scenes, that code is going to trigger some math to find patterns in
3:21
those numbers. If you would like to deep dive into the math behind the code, I'm going to be linking extra resources for that. However, we're going to be getting hands on and writing lots of code to do lots of this. And so if we keep going to break things down a little bit more, machine learning versus deep learning, if we have this giant bubble here of artificial intelligence, you might have seen something similar like this on the internet. I've just
3:48
copied that and put it into pretty colors for this course. So you've got this overarching big bubble of the topic of artificial intelligence, which you could define as, again, almost anything you want. Then typically, there's a subset within artificial intelligence, which is known as machine learning, which is quite a broad topic. And then within machine learning, you have another topic called deep learning. And so that's what we're going to be focused on working with PyTorch,
4:16
writing deep learning code. But again, you could use PyTorch for a lot of different machine learning things. And truth be told, I kind of use these two terms interchangeably. Yes, ML is the broader topic and deep learning is a bit more nuanced. But again, if you want to form your own definitions of these, I'd highly encourage you to do so. This course is more focused on, rather than defining what things are, is seeing how they work. So this is what we're focused on doing.
4:47
Just to break things down, if you're familiar with the fundamentals of machine learning, you probably understand this paradigm, but we're going to just rehash on it anyway. So if we consider traditional programming, let's say you'd like to write a computer program that's enabled to, or has the ability to reproduce your grandmother's favorite or famous roast chicken dish. And so we might have some inputs here, which are some beautiful vegetables, a chicken that you've raised on the
5:17
farm. You might write down some rules. This could be your program, cut the vegetables, season the chicken, preheat the oven, cook the chicken for 30 minutes and add vegetables. Now, it might not be this simple, or it might actually be because your Sicilian grandmother is a great cook. So she's put things into an art now and can just do it step by step. And then those inputs combined with those rules makes this beautiful roast chicken dish. So that's traditional programming. Now,
5:46
a machine learning algorithm typically takes some inputs and some desired outputs and then figures out the rules. So the patterns between the inputs and the outputs. So where in traditional program, we had to hand write all of these rules, the ideal machine learning algorithm will figure out this bridge between our inputs and our idealized output. Now, in the machine learning sense, this is typically described as supervised learning, because you will have some kind of input with
6:18
some kind of output, also known as features, and also known as labels. And the machine learning algorithm's job is to figure out the relationships between the inputs or the features and the outputs or the label. So if we wanted to write a machine learning algorithm to figure out our Sicilian grandmother's famous roast chicken dish, we would probably gather a bunch of inputs of ingredients such as these delicious vegetables and chicken, and then have a whole bunch of outputs of the
6:47
finished product and see if our algorithm can figure out what we should do to go from these inputs to output. So that's almost enough to cover of the difference between traditional programming and machine learning as far as definitions go. We're going to get hands on encoding these sort of algorithms throughout the course. For now, let's go to the next video and ask the question, why use machine learning or deep learning? And actually, before we get there, I'd like you to
7:17
think about that. So going back to what we just saw, the paradigm between traditional programming and machine learning, why would you want to use machine learning algorithms rather than traditional programming? So if you had to write all these rules, could that get cumbersome? So have a think about it and we'll cover it in the next video.
7:41
Welcome back. So in the last video, we covered briefly the difference between traditional programming and machine learning. And again, I don't want to spend too much time on definitions. I'd rather you see this in practice. And I left you with the question, why would you want to use machine learning or deep learning? Well, let's think of a good reason. Why not? I mean, if we had to write all those handwritten rules to reproduce Alsace and grandmother's
8:06
roast chicken dish all the time, that would be quite cumbersome, right? Well, let's draw a line on that. Why not? What's a better reason? And kind of what we just said, right? For a complex problem, can you think of all the rules? So let's imagine we're trying to build a self-driving car. Now, if you've learned to drive, you've probably done so in maybe 20 hours, 100 hours. But now, I'll give you a task of writing down every single rule about driving. How do you back out of your
8:34
driveway? How do you turn left and go down the street? How do you park a reverse park? How do you stop at an intersection? How do you know how fast to go somewhere? So we just listed half a dozen rules. But you could probably go a fair few more. You might get into the thousands. And so for a complex problem, such as driving, can you think of all the rules? Well, probably not. So that's where machine learning and deep learning come in to help. And so this is a beautiful comment
9:02
I like to share with you on one of my YouTube videos is my 2020 machine learning roadmap. And this is from Yashawing. I'm probably going to mispronounce this if I even try to. But Yashawing says, I think you can use ML. So ML is machine learning. I'm going to use that a lot throughout the course, by the way. ML is machine learning, just so you know. For literally anything, as long as you can convert it into numbers, ah, that's what we said before,
9:27
machine learning is turning something into computer readable numbers. And then programming it to find patterns, except with a machine learning algorithm, typically we write the algorithm and it finds the patterns, not us. And so literally it could be anything, any input or output from the universe. That's pretty darn cool about machine learning, right? But should you always use it just because it could be used for anything? Well, I'd like to also introduce you to Google's number one rule
9:58
of machine learning. Now, if you can build a simple rule based system such as the step of five rules that we had to map the ingredients to our Sicilian grandmothers roast chicken dish, if you can write just five steps to do that, that's going to work every time, well, you should probably do that. So if you can build a simple rule based system that doesn't require machine learning, do that. And of course, maybe it's not so very simple, but maybe you can just write some
10:26
rules to solve the problem that you're working on. And this is from a wise software engineer, which is, I kind of hinted at it before, rule one of Google's machine learning handbook. Now, I'm going to highly recommend you read through that, but we're not going to go through that in this video. So check that out. You can Google that otherwise the links will be where you get links. So just keep that in mind, although machine learning is very powerful and very fun and very
10:49
excited, it doesn't mean that you should always use it. I know this is quite the thing to be saying at the start of a deep learning machine learning course, but I just want you to keep in mind, simple rule based systems are still good. Machine learning isn't a solve all for everything. Now, let's have a look at what deep learning is good for, but I'm going to leave you on a clip hammock because we're going to check this out in the next video. See you soon.
11:14
In the last video, we familiarized ourselves with Google's number one rule of machine learning, which is basically if you don't need it, don't use it. And with that in mind, what should we actually be looking to use machine learning or deep learning for?
11:30
Well, problems with long lists of rules. So when the traditional approach fails to remember the traditional approach is you have some sort of data input, you write a list of rules for that data to be manipulated in some way, shape, or form, and then you have the outputs that you know. But if you have a long, long list of rules, like the rules of driving a car, which could be hundreds, could be thousands, could be millions, who knows, that's where machine learning and
11:54
deep learning may help. And it kind of is at the moment in the world of self-driving cars, machine learning and deep learning are the state of the art approach.
12:03
Continually changing environments. So whatever the benefits of deep learning is that it can keep learning if it needs to. And so it can adapt and learn to new scenarios. So if you update the data that your model was trained on, it can adjust to new different kinds of data in the future. So similarly to if you are driving a car, you might know your own neighborhood very well. But then when you go to somewhere you haven't been before, sure you can draw on the foundations
12:30
of what you know, but you're going to have to adapt. How fast should you go? Where should you stop? Where should you park? These kinds of things. So with problems with long lists of rules, or continually changing environments, or if you had a large, large data set. And so this is where deep learning is flourishing in the world of technology. So let's give an example. One of my favorites is the food 101 data set, which you can search for online, which is images of 101
13:00
different kinds of foods. Now we briefly looked at what a rule list might look like for cooking your grandmother's famous Sicilian roast chicken dish. But can you imagine if you wanted to build an app that could take photos of different food, how long your list of rules would be to differentiate 101 different foods? It'd be so long. You need rule sets for every single one. Let's just take one food, for example. How do you write a program to tell what a banana looks like? I mean you'd
13:32
have to code what a banana looks like, but not only a banana, what everything that isn't a banana looks like. So keep this in mind. What deep learning is good for? Problems with long lists of rules, continually changing environments, or discovering insights within large collections of data.
13:48
Now, what deep learning is not good for? And I'm going to write typically here because, again, this is problem specific. Deep learning is quite powerful these days and things might change in the future. So keep an open mind, if there's anything about this course, it's not for me to tell you exactly what's what. It's for me to spark a curiosity into you to figure out what's what, or even better yet, what's not what. So when you need explainability, as we'll see,
14:13
the patterns learned by a deep learning model, which is lots of numbers, called weights and biases, we'll have a look at that later on, are typically uninterpretable by a human. So some of the times deep learning models can have a million, 10 million, 100 million, a billion, some models are getting into the trillions of parameters. When I say parameters, I mean numbers or patterns in data. Remember, machine learning is turning things into numbers and then writing a machine learning model
14:40
to find patterns in those numbers. So sometimes those patterns themselves can be lists of numbers that are in the millions. And so can you imagine looking at a list of numbers that has a million different things going on? That's going to be quite hard. I find it hard to understand three or four numbers, let alone a million. And when the traditional approach is a better option, again, this is Google's rule number one of machine learning. If you can do what you need to do with
15:05
a simple rule based system, well, maybe you don't need to use machine learning or deep learning. Again, I'm going to use the deep learning machine learning terms interchangeably. I'm not too concerned with definitions. You can form your own definitions, but just so you know, from my perspective, ML and deep learning are quite similar. When arrows are unacceptable. So since the outputs of a deep learning model aren't always predictable, we'll see that deep
15:31
learning models are probabilistic. That means they're when they predict something, they're making a probabilistic bet on it. Whereas in a rule based system, you kind of know what the outputs are going to be every single time. So if you can't have errors based on probabilistic errors, well, then you probably shouldn't use deep learning and you'd like to go back to a simple rule based system. And then finally, when you don't have much data, so deep learning models usually require a
15:59
fairly large amount of data to produce great results. However, there's a caveat here, you know, at the start, I said typically, we're going to see some techniques of how to get great results without huge amounts of data. And again, I wrote typically here because there are techniques, you can just research deep learning explainability. You're going to find a whole bunch of stuff. You can look up examples of when machine learning versus deep learning. And then when arrows are
16:24
unacceptable, again, there are ways to make your model reproducible. So it predicts you know what's going to come out. So we do a lot of testing to verify this as well. And so what's next? Ah, we've got machine learning versus deep learning, and we're going to have a look at some different problem spaces in a second, and mainly breaking down in terms of what kind of data you have. Not going to do this now prevent this video from getting too long. We'll cover all these
16:50
colorful beautiful pictures in the next video. Welcome back. So in the last video, we covered a few things of what deep learning is good for and what deep learning is typically not good for. So let's dive in to a little more of a comparison of machine learning versus deep learning. Again, I'm going to be using these terms quite interchangeably. But there are some specific things that typically you want traditional style of machine learning techniques versus deep learning. However,
17:21
this is constantly changing. So again, I'm not talking in absolutes here. I'm more just talking in general. And I'll leave it to you to use your own curiosity to research the specific differences between these two. But typically, for machine learning, like the traditional style of algorithms, although they are still machine learning algorithms, which is kind of a little bit confusing where deep learning and machine learning differ is you want to use traditional
17:49
machine learning algorithms on structured data. So if you have tables of numbers, this is what I mean by structured rows and columns, structured data. And possibly one of the best algorithms for this type of data is a gradient boosted machine, such as xg boost. This is an algorithm that you'll see in a lot of data science competitions, and also used in production settings. When I say production settings, I mean, applications that you may interact with on the internet,
18:17
or use on a day to day. So that's production. xg boost is typically the favorite algorithm for these kinds of situations. So again, if you have structured data, you might look into xg boost rather than building a deep learning algorithm. But again, the rules aren't set in stone. That's where deep learning and machine learning is kind of an art kind of a science is that sometimes xg boost is the best for structured data, but there might be exceptions to the rule. But for deep
18:45
learning, it is typically better for unstructured data. And what I mean by that is data that's kind of all over the place. It's not in your nice, standardized rows and columns. So say you had natural language such as this tweet by this person, whose name is quite similar to mine, and has the same Twitter account as me. Oh, maybe I wrote that. How do I learn machine learning? What you need to hear? Learn Python, learn math, start probability, software engineering, build.
19:12
What you need to do? Google it, go down the rabbit hole, resurfacing six to nine months, and ring assess. I like that. Or if you had a whole bunch of texts such as the definition for deep learning on Wikipedia, again, this is the reason why I'm not covering as many definitions in this course is because look how simple you can look these things up. Wikipedia is going to be able to define deep learning far better than what I can. I'm more focused on just getting involved
19:36
in working hands on with this stuff than defining what it is. And then we have
19:42
images. If we wanted to build a burger, take a photo app thing, you would work with image data, which doesn't really have much of a structure. Although we'll see that there are ways for deep learning that we can turn this kind of data to have some sort of structure through the beauty of a tensor. And then we might have audio files such as if you were talking to your voice assistant. I'm not going to say one because a whole bunch of my devices might go crazy if I say the name of
20:11
my voice assistant, which rhymes with I'm not even going to say that out loud. And so typically, for unstructured data, you'll want to use a neural network of some kind. So structured data, gradient boosted machine, or a random forest, or a tree based algorithm, such as extra boost, and unstructured data, neural networks. So let's keep going. Let's have a look at some of the common algorithms that you might use for structured data, machine learning versus unstructured data,
20:44
deep learning. So random forest is one of my favorites, gradient boosted models, native base nearest neighbor, support vector machine, SVM, and then many more. But since the advent of deep learning, these are often referred to as shallow algorithms. So deep learning, why is it called deep learning? Well, as we'll see is that it can have many different layers of algorithm, you might have an input layer, 100 layers in the middle, and then an output layer.
21:11
But we'll get hands on with this later on. And so common algorithms for deep learning and neural networks, fully connected neural network, convolutional neural network, recurrent neural network, transformers have taken over over the past couple years, and of course, many more. And the beautiful thing about deep learning and neural networks is is almost as many problems that it can be applied to is as many different ways that you can construct them. So this is why I'm putting all these
21:39
dot points on the page. And I can understand if you haven't had much experience of machine learning or deep learning, this can be a whole bunch of information overload. But good news is what we're going to be focused on building with PyTorch is neural networks, fully connected neural networks and convolutional neural networks, the foundation of deep learning. But the excellent thing is, the exciting thing is, is that if we learn these foundational building blocks,
22:03
we can get into these other styles of things here. And again, part art, part science of machine learning and deep learning is depending on how you represent your problem, depending on what your problem is, many of the algorithms here and here can be used for both. So I know I've just kind of bedazzled you and saying that, Oh, well, you kind of use these ones for deep learning, you kind of use these ones for machine learning. But depending on what your problem is, you can also use both.
22:30
So that's a little bit of confusion to machine learning. But that's a fun part about it too, is use your curiosity to figure out what's best for whatever you're working on. And with all this talk about neural networks, how about in the next video, we cover what are neural networks. Now, I'd like you to Google this before we watch the next video, because it's going to be hundreds of definitions of what they are. And I'd like you to start forming your own definition of what a
22:57
neural network is. I'll see you in the next video. Welcome back. In the last video, I left you with the cliffhanger of a question. What are neural networks? And I gave you the challenge of Googling that, but you might have already done that by the time you've got here. Let's just do that together. If I type in what are neural networks, I've already done this. What are neural networks? Explain neural networks, neural network definition. There are hundreds
23:25
of definitions of things like this online neural network in five minutes. Three blue one brown. I'd highly recommend that channel series on neural networks. That's going to be in the extracurricular stat quest is also amazing. So there's hundreds of different definitions on here, and you can read 10 of them, five of them, three of them, make your own definition. But for the sake of this course, here's how I'm going to find neural networks.
23:50
So we have some data of whatever it is. We might have images of food. We might have tweets or natural language, and we might have speech. So these are some examples of inputs for unstructured data, because they're not rows and columns. So these are the input data that we have. And then how do we use them with a neural network? Well, before data can be used in a neural network, it needs to be turned into numbers, because humans, we like looking at images of Raman and
24:18
spaghetti. We know that that's Raman. We know that that's spaghetti after we've seen it one or two times. And we like reading good tweets, and we like listening to amazing music or hearing our friend talk on the phone in audio file. However, before a computer understands what's going on in these inputs, it needs to turn them into numbers. So this is what I call a numerical encoding or a representation. And this numerical encoding, these square brackets indicate that
24:47
it's part of a matrix or a tensor, which we're going to get very hands on with throughout this course. So we have our inputs, we've turned it into numbers, and then we pass it through a neural network. And now this is a graphic for a neural network. However, the graphics for neural networks, as we'll see, can get quite involved. But they all represent the same fundamentals. So if we go to this one, for example, we have an input layer, then we have multiple hidden layers. However,
25:18
you define this, you can design these and how you want. Then we have an output layer. So our inputs will go in some kind of data. The hidden layers will perform mathematical operations on the input. So the numbers, and then we'll have an output. Oh, there's three blue one brown neural networks from the ground up. Great video. Highly recommend you check that out. But then if we come back to this, so we've got our inputs, we've turned it into numbers. And we've got our neural
25:45
networks that we put the input in. This is typically the input layer, hidden layer. This can be as many different layers as you want, as many different, each of these little dots is called a node. There's a lot of information here, but we're going to get hands-on with seeing what this looks like. And then we have some kind of output. Now, which neural network should you use? Well, you can choose the appropriate neural network for your problem, which could involve you
26:07
hand coding each one of these steps. Or you could find one that has worked on problems similar to your own, such as for images, you might use a CNN, which is a convolutional neural network. For natural language, you might use a transformer. For speech, you might also use a transformer. But fundamentally, they all follow the same principle of inputs, manipulation, outputs. And so the neural network will learn a representation on its own. We want to find what it learns.
26:35
So it's going to manipulate these patterns in some way, shape, or form. And when I say learns representation, I'm going to also refer to it as learns patterns in the data. A lot of people refer to it as features. A feature may be the fact that the word do comes out to how, usually, in across a whole bunch of different languages. A feature can be almost anything you want. And again, we don't define this. The neural network learns these representations,
27:02
patterns, features, also called weights on its own. And then where do we go from there? Well, we've got some sort of numbers, numerical encoding turned our data into numbers. Our neural network has learned a representation that it thinks best represents the patterns in our data. And then it outputs those representation outputs, which we can use. And often you'll hear this referred to as features or weight matrix or weight tensor.
27:31
Learned representation is also another common one. There's a lot of different terms for these things. And then it will output. We can convert these outputs into human understandable outputs. So if we were to look at these, this could be, again, I said representations or patterns that are neural network learns can be millions of numbers. This is only nine. So imagine if these were millions of different numbers, I can barely understand the nine numbers that is going on here.
28:00
So we need a way to convert these into human understandable terms. So for this example, we might have some input data, which are images of food. And then we want our neural network to learn the representations between an image of ramen and an image of spaghetti. And then eventually we'll take those patterns that it's learned and we'll convert them into whether it thinks that this is an image of ramen or spaghetti. Or in the case of this tweet,
28:25
is this a tweet for a natural disaster or not a natural disaster? So our neural network has, well, we've written code to turn this into numbers. Pass it through our neural network. Our neural network has learned some kind of patterns. And then we ideally want it to represent this tweet as not a disaster. And then we can write code to do each of these steps here. And the same thing for these inputs going as speech, turning into something that you might say to your smart speaker,
28:54
which I'm not going to say because a whole bunch of my devices might go off. And so let's cover the anatomy of neural networks. We've hinted at this a little bit already. But this is like neural network anatomy 101. Again, this is highly customizable what this thing actually is. We're going to see it in PyTorch code later on. But the data goes into the input layer. And in this case, the number of units slash neurons slash nodes is two hidden layers. You can have, I put a s here
29:26
because you can have one hidden layer, or the deep in deep learning comes from having lots of layers. So this is only showing four layers. You might have, well, this is three layers as well. It might be very deep neural networks such as ResNet 152. This is 152 different layers. So again, you can, or this is 34, because this is only ResNet 34. But ResNet 152 has 152 different layers. So that's a common computer vision or a popular computer vision algorithm, by the way.
30:05
Lots of terms we're throwing out here. But with time, you'll start to become familiar with them. So hidden layers can be almost as many as you want. We've only got pictured one here. And in this case, there's three hidden units slash neurons. And then we have an output layer. So the outputs learned representation or prediction probabilities from here, depending on how we set it up, which again, we will see what these are later on. And in this case, it has one hidden unit. So two input,
30:35
three, one output, you can customize the number of these, you can customize how many layers there are, you can customize what goes into here, you can customize what goes out of there. So now, if we talk about the overall architecture, which is describing all of the layers combined. So that's, when you hear neural network architecture, it talks about the input, the hidden layers, which may be more than one, and the output layer. So that's a terminology for overall architecture.
31:05
Now, I say patterns is an arbitrary term. You can hear embedding embedding might come from hidden layers, weights, feature representation, feature vectors, all referring to similar things. So, again, how do we turn our data into some numerical form, build a neural network to figure out patterns to output some desired output that we want. And now to get more technical, each layer is usually a combination of linear, so straight lines, and nonlinear, non-straight functions. So what I mean by that
31:38
is a linear function is a straight line, a nonlinear function is a non-straight line. If I asked you to draw whatever you want with unlimited straight lines and not straight lines, so you can use straight lines or curved lines, what kind of patterns could you draw?
31:55
At a fundamental level, that is basically what a neural network is doing. It's using a combination of linear, straight lines, and not straight lines to draw patterns in our data. We'll see what this looks like later on. Now, from the next video, let's dive in briefly to different kinds of learning. So we've looked at what a neural network is, the overall algorithm, but there are also different paradigms of how a neural network learns. I'll see you in the next video.
32:22
Welcome back. We've discussed a brief overview of an anatomy of what a neural network is, but let's now discuss some learning paradigms. So the first one is supervised learning, and then we have unsupervised and self-supervised learning, and transfer learning. Now supervised learning is when you have data and labels, such as in the example we gave at the start, which was how you would build a neural network or a machine learning algorithm to figure out the rules to
32:55
cook your Sicilian grandmother's famous roast chicken dish. So in the case of supervised learning, you'd have a lot of data, so inputs, such as raw ingredients as vegetables and chicken, and a lot of examples of what that inputs should ideally look like. Or in the case of discerning photos between a cat and a dog, you might have a thousand photos of a cat and a thousand photos of a dog that you know which photos are cat and which photos are dog, and you pass those photos
33:25
to a machine learning algorithm to discern. So in that case, you have data, the photos, and the labels, aka cat and dog, for each of those photos. So that's supervised learning, data and labels. Unsupervised and self-supervised learning is you just have the data itself. You don't have any labels. So in the case of cat and dog photos, you only have the photos. You don't have the labels of cat and dog. So in the case of self-supervised learning,
33:54
you could get a machine learning algorithm to learn an inherent representation of what, and when I say representation, I mean patterns and numbers, I mean weights, I mean features, a whole bunch of different names describing the same thing. You could get a self-supervised learning algorithm to figure out the fundamental patterns between a dog and a cat image, but it wouldn't necessarily know the difference between the two.
34:19
That's where you could come in later and go show me the patterns you've learned, and it might show you the patterns and you could go, okay, the patterns that look like this, a dog and the patterns that look like that, a cat. So self-supervised and unsupervised learning learn solely on the data itself. And then finally, transfer learning is a very, very important paradigm in deep learning. It's taking the patterns that one model has learned
34:42
of a data set and transferring it to another model, such in the case of if we were trying to build a supervised learning algorithm for discerning between cat and dog photos. We might start with a model that has already learned patterns and images and transfer those foundational patterns to our own model so that our model gets a head start. This is transfer learning is a very, very powerful technique, but as for this course,
35:08
we're going to be writing code to focus on these two supervised learning and transfer learning, which are two of the most common paradigms or common types of learning in machine learning and deep learning. However, this style of code though can be adapted across different learning paradigms. Now, I just want to let you know there is one that I haven't mentioned here, which is kind of in its own bucket, and that is reinforcement learning. So I'll leave this
35:33
as an extension if you wanted to look it up. But essentially, this is a good one. That's a good photo, actually. So shout out to Katie Nuggets. The whole idea of reinforcement learning is that you have some kind of environment and an agent that does actions in that environment, and you give rewards and observations back to that agent. So say, for example, you wanted to teach your dog to urinate outside. Well, you would reward its actions of urinating
36:01
outside and possibly not reward its actions of urinating all over your couch. So reinforcement learning is again, it's kind of in its own paradigm. This picture has a good explanation between unsupervised learning, supervised learning to separate two different things, and then reinforcement learning is kind of like that. But again, I will let you research the different learning paradigms a little bit more in your own time. As I said, we're going to be
36:27
focused on writing code to do supervised learning and transfer learning, specifically pytorch code. Now with that covered, let's get a few examples of what is deep learning actually used for. And before we get into the next video, I'm going to issue you a challenge to search this question yourself and come up with some of your own ideas for what deep learning is currently used for. So give that a shot and I'll see you in the next video. How'd you go? Did you do some research?
36:58
Did you find out what deep learning is actually used for? I bet you found a treasure trail of things. And hey, I mean, if you're reading this course, chances are that you probably already know some use cases for deep learning. You're like, Daniel, hurry up and get to the code. Well, we're going to get there, don't you worry? But let's have a look at some things that deep learning can be used for. But before, I just want to remind you of this comment. This is from
37:20
Yasha Sway on the 2020 machine learning roadmap video. I think you can use ML and remember, ML is machine learning. And remember, deep learning is a part of ML for literally anything as long as you can convert it into numbers and program it to find patterns. Literally, it could be anything, any input or output from the universe. So that's a beautiful thing about machine learning is that if you can encode it something into numbers, chances are you can build a machine learning
37:48
algorithm to find patterns in those numbers. Will it work? Well, again, that's the reason machine learning and deep learning is part art, part science. A scientist would love to know that their experiments would work. But an artist is kind of excited about the fact that, I don't know, this might work, it might not. And so that's something to keep in mind. Along with the rule number one of machine learning is if you don't need it, you don't use it. But if you do use it,
38:12
it can be used for almost anything. And let's get a little bit specific and find out some deep learning use cases. And I've put some up there for a reason because there are lots. These are just some that I interact with in my day to day life, such as recommendation, we've got a programming video, we've got a programming podcast, we got some jujitsu videos, we've got some RuneScape videos, a soundtrack from my favorite movie. Have you noticed, whenever you go to YouTube,
38:38
you don't really search for things anymore. Well, sometimes you might, but the recommendation page is pretty darn good. That's all powered by deep learning. And in the last 10 years, have you noticed that translation has got pretty good too? Well, that's powered by deep learning as well. Now, I don't have much hands on experience with this. I did use it when I was in Japan. I speak a very little amount of Japanese and even smaller amount of Mandarin. But if I wanted to
39:05
translate deep learning as epic to Spanish, it might come out as el aprendise, profando es ebiko. Now, all of the native Spanish speakers watching this video can laugh at me because that was a very Australian version of saying deep learning is epic in Spanish. But that's so cool. All the Google Translate is now powered by deep learning. And the beautiful thing, if I couldn't say it myself, I could click this speaker and it would say it for me. So that speech recognition that's powered
39:37
by deep learning. So if you were to ask your voice assistant who's the biggest big dog of them all, of course, they're going to say you, which is what I've set up, my voice assistant to say. That's part of speech recognition. And in computer vision, oh, look at this. You see this? Where is this photo from? This photo is from this person driving this car. Did a hit and run on my car, at the front of my house, my apartment building, my car was parked on the street, this car, the
40:02
trailer came off, ran into the back of my car, basically destroyed it, and then they drove off. However, my next door neighbors security camera picked up on this car. Now, I became a detective for a week, and I thought, hmm, if there was a computer vision algorithm built into that camera, it could have detected when the car hit. I mean, it took a lot of searching to find it, it turns out the car hit about 3.30am in the morning. So it's pitch black. And of course,
40:28
we didn't get the license plate. So this person is out there somewhere in the world after doing a hit and run. So if you're watching this video, just remember computer vision might catch you one day. So this is called object detection, where you would place a box around the area where the pixels most represent the object that you're looking for. So for computer vision, we could train an object detector to capture cars that drive past a certain camera. And then if someone
40:52
does a hit and run on you, you could capture it. And then fingers crossed, it's not too dark that you can read the license plate and go, hey, excuse me, please, this person has hit my car and wrecked it. So that's a very close to home story of where computer vision could be used. And then finally, natural language processing. Have you noticed as well, your spam detector on your email inbox is pretty darn good? Well, some are powered by deep learning, some not,
41:14
it's hard to tell these days what is powered by deep learning, what isn't. But natural language processing is the process of looking at natural language text. So unstructured text. So whatever you'd write an email in a story in a Wikipedia document and deciding or getting your algorithm to find patterns in that. So for this example, you would find that this email is not spam. This deep learning course is incredible. I can't wait to use what I've learned. Thank you so much.
41:40
And by the way, that is my real email. So if you want to email me, you can. And then this is spam. Hey, Daniel, congratulations, you win a lot of money. Wow, I really like that a lot of money. But somebody said, I don't think that this is real. So that would probably go to my spam inbox. Now, with that being said, if we wanted to put these problems in a little bit more of a classification, this is known as sequence to sequence because you put one sequence in
42:09
and get one sequence out. Same as this, you have a sequence of audio waves and you get some text out. So sequence to sequence, sec to sec. This is classification slash regression. In this case, the regression is predicting a number. That's what a regression problem is. You would predict the coordinates of where these box corners should be. So say this should be at however many pixels in from the X angle and however many pixels down from the Y angle, that's that corner.
42:38
And then you would draw in between the corners. And then the classification part would go, Hey, this is that car that did a hit and run on us. And in this case, this is classification. Classification is predicting whether something is one thing or another, or perhaps more than one thing or another in the class of multi class classification. So this email is not spam. That's a class and this email is spam. So that's also a class. So I think we've only got one direction
43:06
to go now that we've sort of laid the foundation for the course. And that is
43:12
Well, let's start talking about PyTorch. I'll see you in the next video.
43:18
Well, let's now cover some of the foundations of
43:24
PyTorch. But first, you might be asking, what is PyTorch? Well, of course, we could just go to our friend, the internet, and look up PyTorch.org. This is the homepage for PyTorch. This course is not a replacement for everything on this homepage. This should be your ground truth for everything PyTorch. So you can get started. You've got a big ecosystem. You've got a way to set up on your local computer. You've got resources. You've got docs. PyTorch. You've got the GitHub.
43:55
You've got search. You've got blog, everything here. This website should be the place you're visiting most throughout this course as we're writing PyTorch code. You're coming here. You're reading about it. You're checking things out. You're looking at examples. But for the sake of this course, let's break PyTorch down. Oh, there's a little flame animation I just forgot about. What is PyTorch? I didn't sync up the animations. That's all right. So
44:28
PyTorch is the most popular research deep learning framework. I'll get to that in a second. It allows you to write fast deep learning code in Python. If you know Python, it's a very user-friendly programming language. PyTorch allows us to write state-of-the-art deep learning code accelerated by GPUs with Python. It enables you access to many pre-built deep learning models from Torch Hub, which is a website that has lots of, if you remember, I said transfer learning is
45:01
a way that we can use other deep learning models to power our own. Torch Hub is a resource for that. Same as Torch Vision.Models. We'll be looking at this throughout the course. It provides an ecosystem for the whole stack of machine learning. From pre-processing data, getting your data into tenses, what if you started with some images? How do you represent them as numbers? Then you can build models such as neural networks to model that data. Then you can even
45:25
deploy your model in your application slash cloud, well, deploy your PyTorch model. Application slash cloud will be depending on what sort of application slash cloud that you're using, but generally it will run some kind of PyTorch model. And it was originally designed and used in-house by Facebook slash meta. I'm pretty sure Facebook have renamed themselves meta now, but it is now open source and used by companies such as Tesla, Microsoft and OpenAI. And when I say it is the most popular
45:53
deep learning research framework, don't take my word for it. Let's have a look at papers with code dot com slash trends. If you're not sure what papers with code is, it is a website that tracks the latest and greatest machine learning papers and whether or not they have code. So we have some other languages here, other deep learning frameworks, PyTorch, TensorFlow, Jax is another one, MXNet, paddle paddle, the original torch. So PyTorch is an evolution of torch written in Python,
46:18
CAF2, Mindspore. But if we look at this, when is this? Last date is December 2021. We have,
46:26
oh, this is going to move every time I move it. No. So I'll highlight PyTorch at 58% there. So by far and large, the most popular research machine learning framework used to write the code for state of the art machine learning algorithms. So this is browse state of the art papers with code.com amazing website. We have semantic segmentation, image classification, object detection, image generation, computer vision, natural language processing, medical, I'll let you explore this.
46:56
It's one of my favorite resources for staying up to date on the field. But as you see, out of the 65,000 papers with code that this website is tracked, 58% of them are implemented with PyTorch. How cool is that? And this is what we're learning. So let's jump into there. Why PyTorch? Well, other than the reasons that we just spoke about, it's a research favorite. This is highlighting.
47:20
There we go. So there we go. I've highlighted it here. PyTorch, 58%, nearly 2,500 repos. If you're not sure what a repo is, a repo is a place where you store all of your code online. And generally, if a paper gets published in machine learning, if it's fantastic research, it will come with code, code that you can access and use for your own applications or your own research. Again, why PyTorch? Well, this is a tweet from Francois Chale, who's the author of
47:51
Keras, which is another popular deep learning framework. But with tools like Colab, we're going to see what Colab is in a second, Keras and TensorFlow. I've added in here and PyTorch. Virtually anyone can solve in a day with no initial investment problems that would have required an engineering team working for a quarter and $20,000 in hardware in 2014. So this is just to highlight how good the space of deep learning and machine learning tooling has become. Colab,
48:18
Keras and TensorFlow are all fantastic. And now PyTorch is added to this list. If you want to check that out, there's Francois Chale on Twitter. Very, very prominent voice in the machine learning field. Why PyTorch? If you want some more reasons, well, have a look at this. Look at all the places that are using PyTorch. It's just coming up everywhere. We've got Andre Kapathi here, who's the director of AI at Tesla. So if we go, we could search this, PyTorch
48:51
at Tesla. We've got a YouTube talk there, Andre Kapathi, director of AI at Tesla. And so Tesla are using PyTorch for the computer vision models of autopilot. So if we go to videos or maybe images, does it come up there? Things like this, a car detecting what's going on in the scene. Of course, there'll be some other code for planning, but I'll let you research that. When we come back here, OpenAI, which is one of the biggest open artificial intelligence
49:28
research firms, open in the sense that they publish a lot of their research methodologies, however, recently there's been some debate about that. But if you go to openai.com, let's just say that they're one of the biggest AI research entities in the world, and they've standardized on PyTorch. So they've got a great blog, they've got great research, and now they've got OpenAI API, which is, you can use their API to access some of the models
49:56
that they've trained. Presumably with PyTorch, because this blog post from January 2020 says that OpenAI is now standardized across PyTorch. There's a repo called the incredible PyTorch, which collects a whole bunch of different projects that are built on top of PyTorch. That's the beauty of PyTorch is that you can build on top of it, you can build with it AI for AG, for agriculture. PyTorch has been used. Let's have a look. PyTorch in agriculture.
50:22
There we go. Agricultural robots use PyTorch. This is a medium article.
50:31
It's everywhere. So if we go down here, this is using object detection. Beautiful.
50:38
Object detection to detect what kind of weeds should be sprayed with fertilizer. This is just one of many different things, so PyTorch on a big tractor like this. It can be used almost anywhere. If we come back, PyTorch builds the future of AI and machine learning at Facebook, so Facebook, which is also MetaAI, a little bit confusing, even though it says MetaAI, it's on AI.facebook.com. That may change by the time you watch this. They use PyTorch in-house
51:03
for all of their machine learning applications. Microsoft is huge in the PyTorch game. It's absolutely everywhere. So if that's not enough reason to use PyTorch, well, then maybe you're in the wrong course. So you've seen enough reasons of why to use PyTorch. I'm going to give you one more. That is that it helps you run your code, your machine learning code accelerated on a GPU. We've covered this briefly, but what is a GPU slash a TPU,
51:30
because this is more of a newer chip these days. A GPU is a graphics processing unit, which is essentially very fast at crunching numbers. Originally designed for video games, if you've ever designed or played a video game, you know that the graphics are quite intense, especially these days. And so to render those graphics, you need to do a lot of numerical calculations. And so the beautiful thing about PyTorch is that it enables you to leverage a GPU through an
51:56
interface called CUDA, which is a lot of words I'm going to throw out you here, a lot of acronyms in the deep learning space, CUDA. Let's just search CUDA. CUDA toolkit. So CUDA is a parallel computing platform and application programming interface, which is an API that allows software to use certain types of graphics processing units for general purpose computing. That's what we want. So PyTorch leverages CUDA to enable you to run your machine learning code on NVIDIA
52:25
GPUs. Now, there is also an ability to run your PyTorch code on TPUs, which is a tensor processing unit. However, GPUs are far more popular when running various types of PyTorch code. So we're going to focus on running our PyTorch code on the GPU. And to just give you a quick example, PyTorch on TPU, let's see that. Getting started with PyTorch on cloud TPUs, there's plenty of guys for that. But as I said, GPUs are going to be far more common in practice. So that's what
52:57
we're going to focus on. And with that said, we've said tensor processing unit. Now, the reason why these are called tensor processing units is because machine learning and deep learning deals a lot with tensors. And so in the next video, let's answer the question, what is a tensor? But before I go through and answer that from my perspective, I'd like you to research this question. So open up Google or your favorite search engine and type in what is a tensor and
53:27
see what you find. I'll see you in the next video. Welcome back. In the last video, I left you on the cliffhanger question of what is a tensor? And I also issued you the challenge to research what is a tensor. Because as I said, this course isn't all about telling you exactly what things are. It's more so sparking a curiosity in you so that you can stumble upon the answers to these things yourself. But let's have a look. What is a tensor? Now, if you remember this graphic,
53:56
there's a lot going on here. But this is our neural network. We have some kind of input, some kind of numerical encoding. Now, we start with this data. In our case, it's unstructured data because we have some images here, some text here, and some audio file here. Now, these necessarily don't go in all at the same time. This image could just focus on a neural network specifically for images. This text could focus on a neural network specifically for text. And this sound bite
54:23
or speech could focus on a neural network specifically for speech. However, the field is sort of also moving towards building neural networks that are capable of handling all three types of inputs. For now, we're going to start small and then build up the algorithms that we're going to focus on are neural networks that focus on one type of data. But the premise is still the same. You have some kind of input. You have to numerically encode it in some form, pass it to a neural network
54:50
to learn representations or patterns within that numerical encoding, output some form of representation. And then we can convert that representation into things that humans understand. And you might have already seen these, and I might have already referenced the fact that these are tensors. So when the question comes up, what are tensors? A tensor could be almost anything. It could be almost any representation of numbers. We're going to get very hands on with
55:18
tensors. And that's actually the fundamental building block of PyTorch aside from neural network components is the torch dot tensor. We're going to see that very shortly. But this is a very important takeaway is that you have some sort of input data. You're going to numerically encode that data, turn it into a tensor of some kind. Whatever that kind is will depend on the problem you're working with. Then you're going to pass it to a neural network, which will perform mathematical
55:47
operations on that tensor. Now, a lot of those mathematical operations are taken care of by PyTorch behind the scenes. So we'll be writing code to execute some kind of mathematical operations on these tensors. And then the neural network that we create, or the one that's already been created, but we just use for our problem, we'll output another tensor, similar to the input, but has been manipulated in a certain way that we've sort of programmed it to. And then we can take
56:17
this output tensor and change it into something that a human can understand. So to remove a lot of the text around it, make it a little bit more clearer. If we were focusing on building an image classification model, so we want to classify whether this was a photo of Raman or spaghetti, we would have images as input. We would turn those images into numbers, which are represented by a tensor. We would pass that tensor of numbers to a neural network, or there might be lots of
56:45
tensors here. We might have 10,000 images. We might have a million images. Or in some cases, if you're Google or Facebook, you might be working with 300 million or a billion images at a time.
56:56
The principle still stands that you encode your data in some form of numerical representation, which is a tensor, pass that tensor, or lots of tensors to a neural network. The neural network performs mathematical operations on those tensors, outputs a tensor, we convert that tensor into something that we can understand as humans. And so with that being said, we've covered a lot of the fundamentals. What is machine learning? What is deep learning? What is neural network? Well,
57:24
we've touched the surface of these things. You can get as deep as you like. We've covered why use PyTorch. What is PyTorch? Now, the fundamental building block of deep learning is tensors. We've covered that. Let's get a bit more specific in the next video of what we're going to cover code-wise in this first module. I'm so excited we're going to start codes in. I'll see you in the next video. Now it's time to get specific about what we're going to
57:50
cover code-wise in this fundamentals module. But I just want to reiterate the fact that going back to the last video where I challenge you to look up what is a tensor, here's exactly what I would do. I would come to Google. I would type in the question, what is a tensor? There we go. What is a tensor in PyTorch? It knows Google knows that using that deep learning data that we want to know what a tensor is in PyTorch. But a tensor is a very general thing. It's not
58:15
associated with just PyTorch. Now we've got tensor on Wikipedia. We've got tensor. This is probably my favorite video on what is a tensor. By Dan Flesch. Flesch, I'm probably saying that wrong, but good first name. This is going to be your extra curriculum for this video and the previous video is to watch this on what is a tensor. Now you might be saying, well, what gives? I've come to this course to learn PyTorch and all this guy's doing, all you're doing, Daniel, is just Googling
58:43
things when a question comes up. Why don't you just tell me what it is? Well, if I was to tell you everything about deep learning and machine learning and PyTorch and what it is and what it's not, that course would be far too long. I'm doing this on purpose. I'm searching questions like this on purpose because that's exactly what I do day to day as a machine learning engineer. I write code like we're about to do. And then if I don't know something, I literally go to whatever search engine
59:09
I'm using, Google most of the time, and type in whatever error I'm getting or PyTorch, what is a tensor, something like that. So I want to not only tell you that it's okay to search questions like that, but it's encouraged. So just keep that in mind as we go through the whole course, you're going to see me do it a lot. Let's get into what we're going to cover. Here we go.
59:31
Now, this tweet is from Elon Musk. And so I've decided, you know what, let's base the whole course on this tweet. We have learning MLDL from university, you have a little bit of a small brain. Online courses, well, like this one, that brain's starting to explode and you get some little fireworks from YouTube. Oh, you're watching this on YouTube. Look at that shiny brain from articles. My goodness.
59:54
Lucky that this course comes in article format. If you go to learn pytorch.io, all of the course materials are in online book format. So we're going to get into this fundamental section very shortly. But if you want a reference, the course materials are built off this book. And by the time you watch this, there's going to be more chapters here. So we're covering all the bases here. And then finally, from memes, you would ascend to some godlike creature. I think that's
1:00:23
hovering underwater. So that is the best way to learn machine learning. So this is what we're going to start with MLDL from university online courses, YouTube from articles from memes. No, no, no, no. But kind of here is what we're going to cover broadly. So now in this module, we are going to cover the pytorch basics and fundamentals, mainly dealing with tensors and tensor operations. Remember, a neural network is all about input tensors, performing operations on
1:00:52
those tensors, creating output operations. Later, we're going to be focused on pre-processing data, getting it into tensors, so turning data from raw form, images, whatever, into a numerical encoding, which is a tensor. Then we're going to look at building and using pre-trained deep learning models, specifically neural networks. We're going to fit a model to the data. So we're going to show our model or write code for our model to learn patterns in the data that we've
1:01:18
pre-processed. We're going to see how we can make predictions with our model, because that's what deep learning and machine learning is all about, right, using patterns from the past to predict the future. And then we're going to evaluate our model's predictions. We're going to learn how to save and load our models. For example, if you wanted to export your model from where we're working to an application or something like that. And then finally, we're going to see how we can
1:01:39
use a trained model to make predictions on our own data on custom data, which is very fun. And how? Well, you can see that the scientist has faded out a little bit, but that's not really that true. We're going to do it like cooks, not chemists. So chemists are quite precise. Everything has to be exactly how it is. But cooks are more like, oh, you know what, a little bit of salt, a little bit of butter. Does it taste good? Okay, well, then we're on. But machine learning is a little bit of both.
1:02:06
It's a little bit of science, a little bit of art. That's how we're going to do it. But I like the idea of this being a machine learning cooking show. So welcome to cooking with machine learning, cooking with PyTorch with Daniel. And finally, we've got a workflow here, which we have a PyTorch workflow, which is one of many. We're going to kind of use this throughout the entire course is step one, we're going to get our data ready. Step two, we're going to build a
1:02:34
pick a pre trained model to suit whatever problem we're working on. Step two point one, pick a loss function and optimizer. Don't worry about what they are. We're going to cover them soon. Step two point two, build a training loop. Now this is kind of all part of the parcel of step two, hence why we've got two point one and two point two. You'll see what that means later on. Number three, we're going to fit the model to the data and make a prediction. So say we're working
1:02:55
on image classification for Raman or spaghetti. How do we build a neural network or put our images through that neural network to get some sort of idea of what's in an image? We'll see how to do that. Well, the value weight our model to see if it's predicting BS or it's actually going all right. Number five, we're going to improve through experimentation. That's another big thing that you'll notice throughout machine learning throughout this course is that it's
1:03:20
very experimental part art, part science. Number six, save and reload your trained model. Again, I put these with numerical order, but they can kind of be mixed and matched depending on where you are in the journey. But numerical order is just easy to understand for now. Now we've got one more video, maybe another one before we get into code. But in the next video, I'm going to cover some very, very important points on how you should approach this course. I'll see you there.
1:03:54
Now you might be asking, how should I approach this course? You might not be asking, but we're going to answer it anyway. How to approach this course? This is how I would recommend approaching this course. So I'm a machine learning engineer day to day and learning machine learning to coding machine learning, a kind of two different things. I remember when I first learned it was kind of, you learned a lot of theory rather than writing code. So not to take away from the theory
1:04:18
of being important, this course is going to be focusing on writing machine learning specifically PyTorch code. So the number one step to approaching this course is to code along. Now because this course is focused on purely writing code, I will be linking extracurricular resources for you to learn more about what's going on behind the scenes of the code. My idea of teaching is that if we can code together, write some code, see how it's working, that's going to spark your curiosity to
1:04:48
figure out what's going on behind the scenes. So motto number one is if and out, run the code, write it, run the code, see what happens. Number two, I love that. Explore an experiment again. Approach this with the idea, the mind of a scientist and a chef or science and art. Experiment, experiment, experiment. Try things with rigor like a scientist would, and then just try things for the fun of it like a chef would. Number three, visualize what you don't understand. I can't
1:05:25
emphasize this one enough. We have three models so far. If and out, run the code, you're going to hear me say this a lot. Experiment, experiment, experiment. And number three, visualize, visualize, visualize. Why is this? Well, because we've spoken about machine learning and deep learning deals with a lot of data, a lot of numbers. And so I find it that if I visualize some numbers in whatever form that isn't just numbers all over a page, I tend to understand it better.
1:05:51
And there are some great extracurricular resources that I'm going to link that also turn what we're doing. So writing code into fantastic visualizations. Number four, ask questions, including the dumb questions. Really, there's no such thing as a dumb question. Everyone is just on a different part of their learning journey. And in fact, if you do have a quote unquote dumb question, it turns out that a lot of people probably have that one as well. So be sure to ask questions.
1:06:20
I'm going to link a resource in a minute of where you can ask those questions, but please, please, please ask questions, not only to the community, but to Google to the internet to wherever you can, or just yourself. Ask questions of the code and write code to figure out the answer to those questions. Number five, do the exercises. There are some great exercises that I've created for each of the modules. If we go, have we got the book version
1:06:44
of the course up here? We do. Within all of these chapters here, down the bottom is going to be exercises and extra curriculum. So we've got some exercises. I'm not going to jump into them, but I would highly recommend don't just follow along with the course and code after I code. Please, please, please give the exercises a go because that's going to stretch your knowledge.
1:07:09
We're going to have a lot of practice writing code together, doing all of this stuff here. But then the exercises are going to give you a chance to practice what you've learned. And then of course, extra curriculum. Well, hey, if you want to learn more, there's plenty of opportunities to do so there. And then finally, number six, share your work. I can't emphasize enough how much writing about learning deep learning or sharing my work through GitHub or
1:07:36
different code resources or with the community has helped with my learning. So if you learn something cool about PyTorch, I'd love to see it. Link it to me somehow in the Discord chat or on GitHub or whatever. There'll be links of where you can find me. I'd love to see it. Please do share your work. It's a great way to not only learn something because when you share it, when you write about it, it's like, how would someone else understand it? But it's also a great way to
1:08:02
help others learn too. And so we said how to approach this course. Now, let's go how not to approach this course. I would love for you to avoid overthinking the process. And this is your brain, and this is your brain on fire. So avoid having your brain on fire. That's not a good place to be. We are working with PyTorch, so it's going to be quite hot. Just playing on words with the name torch. But avoid your brain catching on fire. And avoid saying, I can't learn,
1:08:33
I've said this to myself lots of times, and then I've practiced it and it turns out I can actually learn those things. So let's just draw a red line on there. Oh, I think a red line. Yeah, there we go. Nice and thick red line. We'll get that out there. It doesn't really make sense now that this says avoid and crossed out. But don't say I can't learn and prevent your brain from catching on fire. Finally, we've got one more video that I'm going to cover before this one
1:08:58
gets too long of the resources for the course before we get into coding. I'll see you there. Now, there are some fundamental resources that I would like you to be aware of before we go any further in this course. These are going to be paramount to what we're working with. So for this course, there are three things. There is the GitHub repo. So if we click this link, I've got a pinned on my browser. So you might want to do the same while you're going through
1:09:25
the course. But this is Mr. D. Burks in my GitHub slash PyTorch deep learning. It is still a work in progress at the time of recording this video. But by the time you go through it, it won't look too much different, but there just be more materials. You'll have materials outline, section, what does it cover? As you can see, some more are coming soon at the time of recording this. So these will probably be done by the time you watch this exercise in extra curriculum.
1:09:49
There'll be links here. Basically, everything you need for the course will be in the GitHub repo. And then if we come back, also on the GitHub repo, the same repo. So Mr. D. Burks slash PyTorch deep learning. If you click on discussions, this is going to be the Q and A. This is just the same link here, the Q and A for the course. So if you have a question here, you can click new discussion, you can go Q and A, and then type in video, and then the title PyTorch Fundamentals, and then go
1:10:25
in here. Or you could type in your error as well. What is N-DIM for a tensor? And then in here, you can type in some stuff here. Hello. I'm having trouble on video X, Y, Z. Put in the name of the video. So that way I can, or someone else can help you out. And then code, you can go three back ticks, write Python, and then you can go import torch, torch dot rand n, which is going to create a tensor. We're going to see this in a second. Yeah, yeah, yeah. And then if you post that
1:10:59
question, the formatting of the code is very helpful that we can understand what's going on, and what's going on here. So this is basically the outline of how I would ask a question video. This is going on. What is such and such for whatever's going on? Hello. This is what I'm having trouble with. Here's the code, and here's what's happening. You could even include the error message, and then you can just click start discussion, and then someone, either myself or someone else from
1:11:26
the course will be able to help out there. And the beautiful thing about this is that it's all in one place. You can start to search it. There's nothing here yet because the course isn't out yet, but as you go through it, there will probably be more and more stuff here. Then if you have any issues with the code that you think needs fixed, you can also open a new issue there. I'll let you read more into what's going on. I've just got some issues here already about the fact that I
1:11:48
need to record videos for the course. I need to create some stuff. But if you think there's something that could be improved, make an issue. If you have a question about the course, ask a discussion. And then if we come back to the keynote, we have one more resource. So that was the course materials all live in the GitHub. The course Q&A is on the course GitHub's discussions tab, and then the course online book. Now, this is a work of art.
1:12:13
This is quite beautiful. It is some code to automatically turn all of the materials from the GitHub. So if we come into here code, if we click on notebook zero zero, this is going to sometimes if you've ever worked with Jupiter notebooks on GitHub, they can take a while to load. So all of the materials here automatically get converted into this book. So the beautiful thing about the book is that it's got different headings here. It's all readable. It's all online.
1:12:40
It's going to have all the images there. And you can also search some stuff here, PyTorch training steps, creating a training loop in PyTorch. Beautiful. We're going to see this later on. So they're the three big materials that you need to be aware of, the three big resources for this specific course materials on GitHub course Q&A course online book, which is learn pytorch.io, simple URL to remember, all the materials will be there. And then
1:13:09
specifically for PyTorch or things PyTorch, the PyTorch website and the PyTorch forums. So if you have a question that's not course related, but more PyTorch related, I'd highly recommend you go to the PyTorch forums, which is available at discuss.pytorch.org. We've got a link there. Then the PyTorch website, PyTorch.org, this is going to be your home ground for everything PyTorch of course. We have the documentation here. And as I said, this course is not a replacement
1:13:36
for getting familiar with the PyTorch documentation. This, the course actually is built off all of the PyTorch documentation. It's just organized in a slightly different way. So there's plenty of amazing resources here on everything to do with PyTorch. This is your home ground. And you're going to see me referring to this a lot throughout the course. So just keep these in mind, course materials on GitHub, course discussions, learnpytorch.io. This is all for the course. And all things
1:14:05
PyTorch specific, so not necessarily this course, but just PyTorch in general, the PyTorch website and the PyTorch forums. With that all being said, we've come so far. We've covered a lot already, but guess what time it is? Let's write some code. I'll see you in the next video. We've covered enough of the fundamentals so far. Well, from a theory point of view, let's get into coding. So I'm going to go over to Google Chrome. I'm going to introduce you to
1:14:36
the tool. One of the main tools we're going to be using for the entire course. And that is Google Colab. So the way I would suggest following along with this course is remember, one of the major ones is to code along. So we're going to go to colab.research.google. I've got a typo here. Classic. You're going to see me do lots of typos throughout this course. Colab.research.google.com. This is going to load up Google Colab. Now, you can follow along with what I'm going to do,
1:15:03
but if you'd like to find out how to use Google Colab from a top-down perspective, you can go through some of these. I'd probably recommend going through overview of Collaboratory Features. But essentially, what Google Colab is going to enable us to do is create a new notebook. And this is how we're going to practice writing PyTorch code. So if you refer to the reference document of learnpytorch.io, these are actually
1:15:30
Colab notebooks just in book format, so online book format. So these are the basis materials for what the course is going to be. There's going to be more here, but every new module, we're going to start a new notebook. And I'm going to just zoom in here. So this one, the first module is going to be zero, zero, because Python code starts at zero, zero. And we're going to call this PyTorch Fundamentals. I'm going to call mine video,
1:15:57
just so we know that this is the notebook that I wrote through the video. And what this is going to do is if we click Connect, it's going to give us a space to write Python code. So here we can go print. Hello, I'm excited to learn PyTorch. And then if we hit shift and enter, it comes out like that. But another beautiful benefit of Google Colab are PS. I'm using the pro version, which costs about $10 a month or so. That price may be different depending on where you're from.
1:16:26
The reason I'm doing that is because I use Colab all the time. However, you do not have to use the paid version for this course. Google Colab comes with a free version, which you'll be able to use to complete this course. If you see it worthwhile, I find the pro version is worthwhile. Another benefit of Google Colab is if we go here, we can go to runtime. Let me just show you that again. Runtime, change runtime type, hardware accelerator. And we can choose to run our code
1:16:55
on an accelerator here. Now we've got GPU and TPU. We're going to be focused on using GPU. If you'd like to look into TPU, I'll leave that to you. But we can click GPU, click save. And now our code, if we write it in such a way, will run on the GPU. Now we're going to see this later on code that runs on the GPU is a lot faster in terms of compute time, especially for deep learning. So if we write here in a video SMI, we now have access to a GPU. In my case, I have a
1:17:27
Tesla P100. It's quite a good GPU. You tend to get the better GPUs. If you pay for Google Colab, if you don't pay for it, you get the free version, you get a free GPU. It just won't be as fast as the GPUs you typically get with the paid version. So just keep that in mind. A whole bunch of stuff that we can do here. I'm not going to go through it all because there's too much. But we've covered basically what we need to cover. So if we just come up here, I'm going to write a text cell. So
1:17:56
oo dot pytorch fundamentals. And I'm going to link in here resource notebook. Now you can come to learn pytorch.io and all the notebooks are going to be in sync. So 00, we can put this in here. Resource notebook is there. That's what this notebook is going to be based off. This one here. And then if you have a question about what's going on in this notebook, you can come to the course GitHub. And then we go back, back. This is where you can see what's
1:18:31
going on. This is pytorch deep learning projects as you can see what's happening. At the moment, I've got pytorch course creation because I'm in the middle of creating it. But if you have a question, you can come to Mr. D Burke slash pytorch deep learning slash discussions, which is this tab here, and then ask a question by clicking new discussion. So any discussions related to this notebook, you can ask it there. And I'm going to turn this right now. This is a code cell.
1:18:56
CoLab is basically comprised of code and text cells. I'm going to turn this into a text cell by pressing command mm, shift and enter. Now we have a text cell. And then if we wanted another code cell, we could go like that text code text code, yada, yada, yada. But I'm going to delete this.
1:19:14
And to finish off this video, we're going to import pytorch. So we're going to import torch. And then we're going to print torch dot dot version. So that's another beautiful thing about Google Colab is that it comes with pytorch pre installed and a lot of other common Python data science packages, such as we could also go import pandas as PD, import NumPy as MP import mapplot lib lib dot pyplot as PLT. This is Google Colab is by far the easiest way to get started with this
1:19:53
course. You can run things locally. If you'd like to do that, I'd refer to you to pytorch deep learning is going to be set up dot MD, getting set up to code pytorch. We've just gone through number one setting up with Google Colab. There is also another option for getting started locally. Right now, this document's a work in progress, but it'll be finished by the time you watch this video. This is not a replacement, though, for the pytorch documentation for getting set up
1:20:20
locally. So if you'd like to run locally on your machine, rather than going on Google Colab, please refer to this documentation or set up dot MD here. But if you'd like to get started as soon as possible, I'd highly recommend you using Google Colab. In fact, the entire course is going to be able to be run through Google Colab. So let's finish off this video, make sure we've got pytorch ready to go. And of course, some fundamental data science packages here.
1:20:47
Wonderful. This means that we have pytorch 1.10.0. So if your version number is far greater than this, maybe you're watching this video a couple of years in the future, and pytorch is up to 2.11, maybe some of the code in this notebook won't work. But 1.10.0 should be more than enough for what we're going to do. And plus Q111, CU111, stands for CUDA version 11.1, I believe. And what that would mean is if we came in here, and we wanted to install it on Linux, which is what
1:21:20
Colab runs on, there's Mac and Windows as well. We've got CUDA. Yeah. So right now, as of recording this video, the latest pytorch build is 1.10.2. So you'll need at least pytorch 1.10 to complete this course and CUDA 11.3. So that's CUDA toolkit. If you remember, CUDA toolkit is NVIDIA's
1:21:44
programming. There we go. NVIDIA developer. CUDA is what enables us to run our pytorch code on NVIDIA GPUs, which we have access to in Google Colab. Beautiful. So we're set up ready to write code. Let's get started in the next video writing some pytorch code. This is so exciting. I'll see you there. So we've got set up. We've got access to pytorch. We've got a Google Colab instance running here. We've got a GPU because we've gone up to runtime, change runtime type, hardware accelerator.
1:22:18
You won't necessarily need a GPU for this entire notebook, but I just wanted to show you how to get access to a GPU because we're going to be using them later on. So let's get rid of this. And one last thing, how I'd recommend going through this course is in a split window fashion. So for example, you might have the video where I'm talking right now and writing code on the left side, and then you might have another window over the other side with your own Colab
1:22:46
window. And you can go new notebook, call it whatever you want, my notebook. You could call it very similar to what we're writing here. And then if I write code over on this side, on this video,
1:23:01
you can't copy it, of course, but you'll write the same code here and then go on and go on and go on. And if you get stuck, of course, you have the reference notebook and you have an opportunity to ask a question here. So with that being said, let's get started. The first thing we're going to have a look at in PyTorch is an introduction to tenses. So tenses are the main building block of deep learning in general, or data. And so you may have watched the video,
1:23:29
what is a tensor? For the sake of this course, tenses are a way to represent data, especially multi dimensional data, numeric data that is, but that numeric data represents something else. So let's go in here, creating tenses. So the first kind of tensor we're going to create is actually called a scalar. I know I'm going to throw a lot of different names of things at you, but it's important that you're aware of such nomenclature. Even though in PyTorch, almost
1:23:58
everything is referred to as a tensor, there are different kinds of tenses. And just to exemplify the fact that we're using a reference notebook, if we go up here, we can see we have importing PyTorch. We've done that. Now we're up to introduction to tenses. We've got creating tenses, and we've got scalar, etc, etc, etc. So this is what we're going to be working through. Let's do it together. So scalar, the way to, oops, what have I done there? The way to create a
1:24:29
tensor in PyTorch, we're going to call this scalar equals torch dot tensor. And we're going to fill it with the number seven. And then if we press or retype in scalar, what do we get back? Seven, wonderful. And it's got the tensor data type here. So how would we find out about what torch dot tensor actually is? Well, let me show you how I would. We go to torch dot tensor. There we go. We've got the documentation. So this is possibly the most common class in PyTorch other than
1:25:00
one we're going to see later on that you'll use, which is torch dot nn. Basically, everything in PyTorch works off torch dot tensor. And if you'd like to learn more, you can read through here. In fact, I would encourage you to read through this documentation for at least 10 minutes after you finish some videos here. So with that being said, I'm going to link that in here. So PyTorch tensors are created using torch dot tensor. And then we've got that link there.
1:25:32
Oops, typos got law Daniel. Come on. They're better than this. No, I'm kidding. There's going to be typos got law through the whole course. Okay. Now, what are some attributes of a scalar? So some details about scalars. Let's find out how many dimensions there are. Oh, and by the way, this warning, perfect timing. Google Colab will give you some warnings here, depending on whether you're using a GPU or not. Now, the reason being is because Google Colab provides GPUs to you and
1:26:01
I for free. However, GPUs aren't free for Google to provide. So if we're not using a GPU, we can save some resources, allow someone else to use a GPU by going to none. And of course, we can always switch this back. So I'm going to turn my GPU off so that someone else out there, I'm not using the GPU at the moment, they can use it. So what you're also going to see is if your Google Colab instance ever restarts up here, we're going to have to rerun these cells. So if
1:26:33
you stop coding for a while, go have a break and then come back and you start your notebook again, that's one downside of Google Colab is that it resets after a few hours. How many hours? I don't know exactly. The reset time is longer if you have the pro subscription, but because it's a free service and the way Google calculate usage and all that sort of stuff, I can't give a conclusive evidence or conclusive answer on how long until it resets. But just know, if you come back, you might
1:27:02
have to rerun some of your cells and you can do that with shift and enter. So a scalar has no dimensions. All right, it's just a single number. But then we move on to the next thing. Or actually, if we wanted to get this number out of a tensor type, we can use scalar dot item, this is going to give it back as just a regular Python integer. Wonderful, there we go, the number seven back, get tensor back as Python int. Now, the next thing that we have is a vector. So let's write
1:27:37
in here vector, which again is going to be created with torch dot tensor. But you will also hear the word vector used a lot too. Now, what is the deal? Oops, seven dot seven. Google Colab's auto complete is a bit funny. It doesn't always do the thing you want it to. So if we see a vector, we've got two numbers here. And then if we really wanted to find out what is a vector. So a vector usually has magnitude and direction. So what we're going to see later on is, there we
1:28:10
go, magnitude, how far it's going and which way it's going. And then if we plotted it, we've got, yeah, a vector equals the magnitude would be the length here and the direction would be where it's pointing. And oh, here we go, scalar vector matrix tensor. This is what we're working on as well. So the thing about vectors, how they differ with scalars is how I just remember them is rather than magnitude and direction is a vector typically has more than one number.
1:28:38
So if we go vector and dim, how many dimensions does it have?
1:28:44
It has one dimension, which is kind of confusing. But when we see tensors with more than one dimension, it'll make sense. And another way that I remember how many dimensions something has is by the number of square brackets. So let's check out something else. Maybe we go vector dot shape shape is two. So the difference between dimension. So dimension is like number of square brackets. And when I say, even though there's two here, I mean number of pairs of closing square
1:29:18
brackets. So there's one pair of closing square brackets here. But the shape of the vector is two. So we have two by one elements. So that means a total of two elements. Now if we wanted to step things up a notch, let's create a matrix. So this is another term you're going to hear. And you might be wondering why I'm capitalizing matrix. Well, I'll explain that in the second matrix equals torch dot tensor. And we're going to put two square brackets here. You might be
1:29:50
thinking, what could the two square brackets mean? Or actually, that's a little bit of a challenge. If one pair of square brackets had an endem of one, what will the endem be number of dimensions of two square brackets? So let's create this matrix. Beautiful. So we've got another tensor here. Again, as I said, these things have different names, like the traditional name of scalar, vector matrix, but they're all still a torch dot tensor. That's a little bit confusing,
1:30:24
but the thing you should remember in PyTorch is basically anytime you encode data into numbers, it's of a tensor data type. And so now how many n number of dimensions do you think a matrix has?
1:30:38
It has two. So there we go. We have two square brackets. So if we wanted to get matrix, let's index on the zeroth axis. Let's see what happens there. Ah, so we get seven and eight. And then we get off the first dimension. Ah, nine and 10. So this is where the square brackets, the pairings come into play. We've got two square bracket pairings on the outside here. So we have an endem of two. Now, if we get the shape of the matrix, what do you think the shape will be?
1:31:08
Ah, two by two. So we've got two numbers here by two. So we have a total of four elements in there.
1:31:22
So we're covering a fair bit of ground here, nice and quick, but that's going to be the teaching style of this course is we're going to get quite hands on and writing a lot of code and just interacting with it rather than continually going back over and discussing what's going on here. The best way to find out what's happening within a matrix is to write more code that's similar to these matrices here. But let's not stop at matrix. Let's upgrade to a tensor now. So I might
1:31:48
put this in capitals as well. And I haven't explained what the capitals mean yet, but we'll see that in a second. So let's go torch dot tensor. And what we're going to do is this time, we've done one square bracket pairing. We've done two square bracket pairings. Let's do three square bracket pairings and just get a little bit adventurous. All right. And so you might be thinking at the moment, this is quite tedious. I'm just going to write a bunch of random numbers here. One,
1:32:16
two, three, three, six, nine, two, five, four. Now you might be thinking, Daniel, you've said tensors could have millions of numbers. If we had to write them all by hand, that would be quite tedious. And yes, you're completely right. The fact is, though, that most of the time, you won't be crafting tensors by hand. PyTorch will do a lot of that behind the scenes. However, it's important to know that these are the fundamental building blocks of the models
1:32:45
and the deep learning neural networks that we're going to be building. So tensor capitals as well, we have three square brackets. So, or three square bracket pairings. I'm just going to refer to three square brackets at the very start because they're going to be paired down here. How many n dim or number of dimensions do you think our tensor will have? Three, wonderful. And what do you think the shape of our tensor is? We have three elements here. We have three elements here, three elements
1:33:17
here. And we have one, two, three. So maybe our tensor has a shape of one by three by three. Hmm. What does that mean? Well, we've got three by one, two, three. That's the second square bracket there by one. Ah, so that's the first dimension there or the zeroth dimension because we remember PyTorch is zero indexed. We have, well, let's just instead of talking about it, let's just get on the zeroth axis and see what happens with the zeroth dimension. There we go.
1:33:54
Okay. So there's, this is the far left one, zero, which is very confusing because we've got a one here, but so we've got, oops, don't mean that. What this is saying is we've got one three by three shape tensor. So very outer bracket matches up with this number one here. And then this three matches up with the next one here, which is one, two, three. And then this three matches up with this one, one, two, three. Now, if you'd like to see this with a pretty picture, we can see it here.
1:34:36
So dim zero lines up. So the blue bracket, the very outer one, lines up with the one. Then dim equals one, this one here, the middle bracket, lines up with the middle dimension here. And then dim equals two, the very inner lines up with these three here. So again, this is going to take a lot of practice. It's taken me a lot of practice to understand the dimensions of tensors. But to practice, I would like you to write out your own tensor of, you can put however many square
1:35:14
brackets you want. And then just interact with the end dim shape and indexing, just as I've done here, but you can put any combination of numbers inside this tensor. That's a little bit of practice before the next video. So give that a shot and then we'll move on to the next topic. I'll see you there. Welcome back. In the last video, we covered the basic building blocks of data representation in deep learning, which is the tensor, or in PyTorch, specifically torch.tensor.
1:35:45
But within that, we had to look at what a scalar is. We had to look at what a vector is. We had to look at a matrix. We had to look at what a tensor is. And I issued you the challenge to get as creative as you like with creating your own tensor. So I hope you gave that a shot because as you'll see throughout the course and your deep learning journey, a tensor can represent or can be of almost any shape and size and have almost any combination of numbers within it. And so this is very important
1:36:13
to be able to interact with different tensors to be able to understand what the different names of things are. So when you hear matrix, you go, oh, maybe that's a two dimensional tensor. When you hear a vector, maybe that's a one dimensional tensor. When you hear a tensor, that could be any amount of dimensions. And just for reference for that, if we come back to the course reference, we've got a scalar. What is it? A single number, number of dimensions, zero. We've got a vector,
1:36:38
a number with direction, number of dimensions, one, a matrix, a tensor. And now here's another little tidbit of the nomenclature of things, the naming of things. Typically, you'll see a variable name for a scalar or a vector as a lowercase. So a vector, you might have a lowercase y storing that data. But for a matrix or a tensor, you'll often see an uppercase letter or variable in Python in our case, because we're writing code. And so I am not exactly sure why this is, but this is just
1:37:12
what you're going to see in machine learning and deep learning code and research papers across the board. This is a typical nomenclature. Scalars and vectors, lowercase, matrix and tensors, uppercase, that's where that naming comes from. And that's why I've given the tensor uppercase here. Now, with that being said, let's jump in to another very important concept with tensors. And that is random tensors. Why random tensors? I'm just writing this in a code cell now.
1:37:40
I could go here. This is a comment in Python, random tensors. But we'll get rid of that. We could just start another text cell here. And then three hashes is going to give us a heading, random tensors there. Or I could turn this again into a markdown cell with command mm when I'm using Google Colab. So random tensors. Let's write down here. Why random tensors? So we've done the tedious thing of creating our own tensors with some numbers that we've defined, whatever these are. Again,
1:38:15
you could define these as almost anything. But random tensors is a big part in pytorch because let's write this down. Random tensors are important because the way many neural networks learn is that they start with tensors full of random numbers and then adjust those random numbers to better represent the data. So seriously, this is one of the big concepts of neural networks. I'm going to write in code here, which is this is what the tick is for. Start with random numbers.
1:38:58
Look at data, update random numbers. Look at data, update random numbers. That is the crux of neural networks. So let's create a random tensor with pytorch. Remember how I said that pytorch is going to create tensors for you behind the scenes? Well, this is one of the ways that it does so. So we create a random tensor and we give it a size of random tensor of size or shape. Pytorch use these independently. So size, shape, they mean the different versions of the same thing.
1:39:47
So random tensor equals torch dot rand. And we're going to type in here three, four. And the beautiful thing about Google Colab as well is that if we wait long enough, it's going to pop up with the doc string of what's going on. I personally find this a little hard to read in Google Colab, because you see you can keep going down there. You might be able to read that. But what can we do? Well, we can go to torch dot rand. Then we go to the documentation. Beautiful. Now there's a whole
1:40:19
bunch of stuff here that you're more than welcome to read. We're not going to go through all that. We're just going to see what happens hands on. So we'll copy that in here. And write this in notes, torch random tensors. Done. Just going to make some code cells down here. So I've got some space. I can get this a bit up here. Let's see what our random tensor looks like. There we go. Beautiful of size three, four. So we've got three or four elements here. And then we've got three deep
1:40:53
here. So again, there's the two pairs. So what do you think the number of dimensions will be for random tensor? And dim. Two beautiful. And so we have some random numbers here. Now the beautiful thing about pie torch again is that it's going to do a lot of this behind the scenes. So if we wanted to create a size of 10 10, in some cases, we won't want one dimension here. And then it's going to go 10 10. And then if we check the number of dimensions, how many do you think it
1:41:24
will be now three? Why is that? Because we've got one 10 10. And then if we wanted to create 10 10 10.
1:41:32
What's the number of dimensions going to be? It's not going to change. Why is that?
1:41:36
We haven't run that cell yet, but we've got a lot of numbers here.
1:41:42
We can find out what 10 times 10 times 10 is. And I know we can do that in our heads, but the beauty of collab is we've got a calculator right here. 10 times 10 times 10. We've got a thousand elements in there. But sometimes tenses can be hundreds of thousands of elements or millions of elements. But pie torch is going to take care of a lot of this behind the scenes. So let's clean up a bit of space here. This is a random tensor. Random numbers beautiful of now
1:42:10
it's got two dimensions because we've got three by four. And if we put another one in the front there, we're going to have how many dimensions three dimensions there. But again, this number of dimensions could be any number. And what's inside here could be any number. Let's get rid of that. And let's get a bit specific because right now this is just a random tensor of whatever dimension. How about we create a random tensor with similar shape to an image tensor. So a lot of the time
1:42:44
when we turn images, image size tensor, when we turn images into tenses, they're going to have, let me just write it in code for you first, size equals a height, a width, and a number of color channels. And so in this case, it's going to be height with color channels. And the color channels are red, green, blue. And so let's create a random image tensor. Let's view the size of it or the shape. And then random image size tensor will view the end dim. Beautiful. Okay, so we've got
1:43:28
torch size, the same size two, two, four, two, four, three, height, width, color channels. And we've got three dimensions, one, four, height, width, color channels. Let's go and see an example of this. This is the PyTorch Fundamentals notebook. If we go up to here, so say we wanted to encode this image of my dad eating pizza with thumbs up of a square image of two, two, four by two, two, four. This is an input. And if we wanted to encode this into tensor format, well, one of the ways of
1:44:02
representing an image tensor, very common ways is to split it into color channels because with red, green, and blue, you can create almost any color you want. And then we have a tensor representation. So sometimes you're going to see color channels come first. We can switch this around and our code quite easily by going color channels here. But you'll also see color channels come at the end. I know I'm saying a lot that we kind of haven't covered yet. The main takeaway
1:44:28
from here is that almost any data can be represented as a tensor. And one of the common ways to represent images is in the format color channels, height, width, and how these values are will depend on what's in the image. But we've done this in a random way. So the takeaway from this video is that PyTorch enables you to create tensors quite easily with the random method. However, it is going to do a lot of this creating tensors for you behind the scenes and why a random tensor is so
1:45:02
valuable because neural networks start with random numbers, look at data such as image tensors, and then adjust those random numbers to better represent that data. And they repeat those steps onwards and onwards and onwards. Let's finish this video here. I'm going to challenge for you just to create your own random tensor of whatever size and shape you want. So you could have 5, 10, 10 here and see what that looks like. And then we'll keep coding in the next video.
1:45:33
I hope you took on the challenge of creating random tensor of your own size. And just a little tidbit here. You might have seen me in the previous video. I didn't use the size parameter. But in this case, I did here, you can go either way. So if we go torch dot rand size equals, we put in a tuple here of three three, we've got that tensor there three three. But then also if we don't put the size in there, it's the default. So it's going to create a very similar tensor. So whether you
1:46:01
have this size or not, it's going to have quite a similar output depending on the shape that you put in there. But now let's get started to another kind of tensor that you might see zeros and ones.
1:46:16
So say you wanted to create a tensor, but that wasn't just full of random numbers, you wanted to create a tensor of all zeros. This is helpful for if you're creating some form of mask. Now, we haven't covered what a mask is. But essentially, if we create a tensor of all zeros,
1:46:41
what happens when you multiply a number by zero? All zeros. So if we wanted to multiply these two together, let's do zeros times random tensor. There we go, all zeros. So maybe if you're working with this random tensor and you wanted to mask out, say all of the numbers in this column for some reason, you could create a tensor of zeros in that column, multiply it by your target tensor, and you would zero all those numbers. That's telling
1:47:15
your model, hey, ignore all of the numbers that are in here because I've zeroed them out. And then if you wanted to create a tensor of all ones, create a tensor of all ones, we can go ones equals torch dot ones, size equals three, four. And then if we have a look, there's another parameter I haven't showed you yet, but this is another important one is the D type. So the default data type, so that's what D type stands for, is torch dot float. We've actually been using torch dot float
1:47:49
the whole time, because that's whenever you create a tensor with pytorch, we're using a pytorch method, unless you explicitly define what the data type is, we'll see that later on, defining what the data type is, it starts off as torch float 32. So these are float numbers. So that is how you create zeros and ones zeros is probably I've seen more common than ones in use, but just keep these in mind, you might come across them. There are lots of different methods to creating
1:48:17
tensors. And truth be told, like random is probably one of the most common, but you might see zeros and ones out in the field. So now we've covered that. Let's move on into the next video, where we're going to create a range. So have a go at creating a tensor full of zeros and whatever size you want, and a tensor full of ones and whatever size you want. And I'll see you in the next video.
1:48:42
Welcome back. I hope you took on the challenge of creating a torch tensor of zeros of your own size and ones of your own size. But now let's investigate how we might create a range of tensors and tensors like. So these are two other very common methods of creating tensors. So let's start by creating a range. So we'll first use torch dot range, because depending on when you're watching this video, torch dot range may be still in play or it may be deprecated.
1:49:19
If we write in torch dot range right now with the pie torch version that I'm using, which is torch dot version, which is torch or pie torch 1.10 point zero torch range is deprecated and will be removed in a future release. So just keep that in mind. If you come across some code that's using torch dot range, maybe out of whack. So the way to get around that is to fix that is to use a range instead. And if we just write in torch dot a range, we've got tensors of zero to nine,
1:49:52
because it of course starts at zero index. If we wanted one to 10, we could go like this. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. And we can go zero, or we go 1, 2, 10, equals torch a range.
1:50:10
Wonderful. And we can also define the step. So let's let's type in some start and where can we find the documentation on a range? Sometimes in Google Colab, you can press shift tab, but I find that it doesn't always work for me. Yeah, you could hover over it, but we can also just go torch a range and look for the documentation torch a range. So we've got start and step. Let's see what all of these three do. Maybe we start at zero, and maybe we want it to go to a thousand,
1:50:44
and then we want a step of what should our step be? What's a fun number? 77. So it's not one to 10 anymore, but here we go. We've got start at zero, 77 plus 77 plus 77, all the way up to it finishes at a thousand. So if we wanted to take it back to one to 10, we can go up here. 110, and the default step is going to be one. Oops, we needed the end to be that it's going to finish at end minus one. There we go. Beautiful. Now we can also create tensors like. So creating tensors like. So tensors
1:51:27
like is say you had a particular shape of a tensor you wanted to replicate somewhere else, but you didn't want to explicitly define what that shape should be. So what's the shape of one to 10?
1:51:42
One to 10. Now if we wanted to create a tensor full of zeros that had the same shape as this, we can use tensor like or zeros like. So 10 zeros, zeros equals, I'm not even sure if I'm spelling zeros right then, zeros. Well, I might have a typo spelling zeros here, but you get what I'm saying is torch zeros. Oh, torch spell it like that. That's why I'm spelling it like that. Zeros like one to 10. And then the input is going to be one to 10. And we have a look at 10 zeros.
1:52:19
My goodness, this is taking quite the while to run. This is troubleshooting on the fly. If something's happening like this, you can try to stop. If something was happening like that, you can click run and then stop. Well, it's running so fast that I can't click stop. If you do also run into trouble, you can go runtime, restart runtime. We might just do that now just to show you. Restart and run all is going to restart the compute engine behind the collab notebook.
1:52:48
And run all the cells to where we are. So let's just see that we restart and run runtime. If you're getting errors, sometimes this helps. There is no set in stone way to troubleshoot errors. It's guess and check with this. So there we go. We've created 10 zeros, which is torch zeros like our one to 10 tensor. So we've got zeros in the same shape as one to 10. So if you'd like to create tensors, use torch arrange and get deprecated message. Use torch arrange instead for creating
1:53:26
a range of tensors with a start and end in a step. And then if you wanted to create tensors or a tensor like something else, you want to look for the like method. And then you put an input, which is another tensor. And then it'll create a similar tensor with whatever this method here is like in that fashion or in the same shape as your input. So with that being said, give that a try, create a range of tensors, and then try to replicate that range shape that you've
1:53:54
made with zeros. I'll see you in the next video. Welcome back. Let's now get into a very important topic of tensor data types. So we've briefly hinted on this before. And I said that let's create a tensor to begin with float 32 tensor. And we're going to go float 32 tensor equals torch dot tensor. And let's just put in the numbers three, six, nine. If you've ever played need for speed underground, you'll know where three, six, nine comes from. And then we're going to go
1:54:34
D type equals, let's just put none and see what happens, hey, float 32 tensor. Oh, what is the data type? float 32, tensor dot D type. float 32, even though we put none, this is because the default data type in pytorch, even if it's specified as none is going to come out as float 32. What if we wanted to change that to something else? Well, let's type in here float 16.
1:55:07
And now we've got float 32 tensor. This variable name is a lie now because it's a float 16 tensor. So we'll leave that as none. Let's go there. There's another parameter when creating tensors. It's very important, which is device. So we'll see what that is later on. And then there's a final one, which is also very important, which is requires grad equals false. Now this could be true, of course, we're going to set this as false. So these are three of the most important parameters
1:55:38
when you're creating tensors. Now, again, you won't necessarily always have to enter these when you're creating tensors, because pytorch does a lot of tensor creation behind the scenes for you. So let's just write out what these are. Data type is what data type is the tensor, e.g. float 32, or float 16. Now, if you'd like to look at what data types are available for pytorch tensors, we can go torch tensor and write up the top unless the documentation changes. We have data types.
1:56:11
It's so important that data types is the first thing that comes up when you're creating a tensor. So we have 32-bit floating point, 64-bit floating point, 16, 16, 32-bit complex. Now, the most common ones that you will likely interact with are 32-bit floating point and 16-bit floating point. Now, what does this mean? What do these numbers actually mean? Well, they have to do with precision in computing. So let's look up that. Precision in computing. Precision computer science.
1:56:44
So in computer science, the precision of a numerical quantity, we're dealing with numbers, right? As a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a value. So, for us, precision is the numerical quantity, is a measure of the detail, how much detail in which the quantity is expressed. So, I'm not going
1:57:14
to dive into the background of computer science and how computers represent numbers. The important takeaway for you from this will be that single precision floating point is usually called float 32, which means, yeah, a number contains 32 bits in computer memory. So if you imagine, if we have a tensor that is using 32 bit floating point, the computer memory stores the number as 32 bits. Or if it has 16 bit floating point, it stores it as 16 bits or 16 numbers representing or 16.
1:57:46
I'm not sure if a bit equates to a single number in computer memory. But what this means is that a 32 bit tensor is single precision. This is half precision. Now, this means that it's the default of 32, float 32, torch dot float 32, as we've seen in code, which means it's going to take up a certain amount of space in computer memory. Now, you might be thinking, why would I do anything other than the default? Well, if you'd like to sacrifice some detail in how your number is
1:58:16
represented. So instead of 32 bits, it's represented by 16 bits, you can calculate faster on numbers that take up less memory. So that is the main differentiator between 32 bit and 16 bit. But if you need more precision, you might go up to 64 bit. So just keep that in mind as you go forward. Single precision is 32. Half precision is 16. What do these numbers represent? They represent how much detail a single number is stored in memory. That was a lot to take in. But we're talking
1:58:53
about 10 to data types. I'm spending a lot of time here, because I'm going to put a note here, note, tensor data types is one of the three big issues with pytorch and deep learning or not not issues, they're going to be errors that you run into and deep learning. Three big errors, you'll run into with pytorch and deep learning. So one is tensors, not right data type. Two tensors, not right shape. We've seen a few shapes of four and three tensors, not on the right
1:59:39
device. And so in this case, if we had a tensor that was float 16 and we were trying to do computations with a tensor that was float 32, we might run into some errors. And so that's the tensors not being in the right data type. So it's important to know about the D type parameter here. And then tensors not being the right shape. Well, that's once we get onto matrix multiplication, we'll see that if one tensor is a certain shape and another tensor is another shape and those shapes don't
2:00:08
line up, we're going to run into shape errors. And this is a perfect segue to the device. Device equals none. By default, this is going to be CPU. This is why we are using Google Colab because it enables us to have access to, oh, we don't want to restart, enables us to have access to a GPU. As I've said before, a GPU enables us. So we could change this to CUDA. That would be, we'll see how to write device agnostic code later on. But this device, if you try to do
2:00:39
operations between two tensors that are not on the same device. So for example, you have one tensor that lives on a GPU for fast computing, and you have another tensor that lives on a CPU and you try to do something with them, while pytorch is going to throw you an error. And then finally, this last requirement is grad is if you want pytorch to track the gradients, we haven't covered what that is of a tensor when it goes through certain numerical calculations. This is a bit of
2:01:06
a bombardment, but I thought I'd throw these in as important parameters to be aware of since we're discussing data type. And really, it would be reminiscent of me to discuss data type without discussing not the right shape or not the right device. So with that being said, let's write down here what device is your tensor on, and whether or not to track gradients with this tensor's operations. So we have a float 32 tensor. Now, how might we change the tensor data type of this?
2:01:44
Let's create float 16 tensor. And we saw that we could explicitly write in float 16 tensor. Or we can just type in here, float 16 tensor equals float 32 tensor dot type. And we're going to type in torch dot float 16, why float 16, because well, that's how we define float 16, or we could use half. So the same thing, these things are the same, let's just do half, or float 16 is more explicit for me. And then let's check out float 16 tensor. Beautiful, we've converted our float
2:02:25
32 tensor into float 16. So that is one of the ways that you'll be able to tackle the tensors not in the right data type issue that you run into. And just a little note on the precision and computing, if you'd like to read more on that, I'm going to link this in here. And this is all about how computers store numbers. So precision in computing. There we go. I'll just get rid of that. Wonderful. So give that a try, create some tensors, research, or go to the documentation of torch
2:02:59
dot tensor and see if you can find out a little bit more about D type device and requires grad, and create some tensors of different data types. Play around with whatever the ones you want here, and see if you can run into some errors, maybe try to multiply two tensors together. So if you go float 16 tensor times float 32 tensor, give that a try and see what happens. I'll see you in the next video. Welcome back. In the last video, we covered a little bit about tensor data types,
2:03:30
as well as some of the most common parameters you'll see past to the torch dot tensor method. And so I should do the challenge at the end of the last video to create some of your own tensors of different data types, and then to see what happens when you multiply a float 16 tensor by a float 32 tensor. Oh, it works. And but you've like Daniel, you said that you're going to have tensors not the right data type. Well, this is another kind of gotcha or caveat of pie torch and deep
2:03:58
learning in general, is that sometimes you'll find that even if you think something may error because these two tensors are different data types, it actually results in no error. But then sometimes you'll have other operations that you do, especially training large neural networks, where you'll get data type issues. The important thing is to just be aware of the fact that some operations will run an error when your tensors are not in the right data type. So let's try another
2:04:23
type. Maybe we try a 32 bit integer. So torch dot in 32. And we try to multiply that by a float. Wonder what will happen then? So let's go into 32 in 32 tensor equals torch dot tensor. And we'll just make it three. Notice that there's no floats there or no dot points to make it a float. Three, six, nine and D type can be torch in 32. And then in 32 tensor, what does this look like? Typo, of course, one of many in 32 tensor. So now let's go float 32 tensor and see what happens.
2:05:12
Can we get pie torch to throw an error in 32 tensor? Huh, it worked as well. Or maybe we go into 64. What happens here?
2:05:28
Still works. Now, see, this is again one of the confusing parts of doing tensor operations. What if we do a long tensor? Torch to long. Is this going to still work?
2:05:41
Ah, torch has no attribute called long. That's not a data type issue. I think it's long tensor. Long tensor. Does this work? D type must be torch D type.
2:05:58
Torch long tensor. I could have sworn that this was torch dot tensor.
2:06:08
Oh, there we go. Torch dot long tensor. That's another word for 64 bit. So what is this saying? CPU tensor. Okay, let's see. This is some troubleshooting on the fly here.
2:06:23
Then we multiply it. This is a float 32 times a long. It works. Okay, so it's actually a bit more robust than what I thought it was. But just keep this in mind when we're training models, we're probably going to run into some errors at some point of our tensor's not being the right data type. And if pie torch throws us an error saying your tensors are in the wrong data type, well, at least we know now how to change that data type or how to set the data type if we
2:06:46
need to. And so with that being said, let's just formalize what we've been doing a fair bit already. And that's getting information from tensors. So the three big things that we'll want to get from our tensors in line with the three big errors that we're going to face in neural networks and deep lining is let's copy these down. Just going to get this, copy this down below. So if we want to get some information from tensors, how do we check the shape? How do we check the data type?
2:07:18
How do we check the device? Let's write that down. So to get information from this, to get D type or let's write data type from a tensor can use tensor dot D type. And let's go here to get shape from a tensor can use tensor dot shape. And to get device from a tensor, which devices it on CPU or GPU can use tensor dot device. Let's see these three in action. So if we run into one of the three big problems in deep learning and neural networks in general, especially with PyTorch,
2:08:07
tensor's not the right data type, tensor's not the right shape or tensor's not on the right device. Let's create a tensor and try these three out. We've got some tensor equals torch dot rand and we'll create it a three four. Let's have a look at what it looks like.
2:08:25
There we go. Random numbers of shape three and four. Now let's find out some details about it.
2:08:32
Find out details about some tensor. So print or print some tensor. And oops, didn't want that print. And let's format it or make an F string of shape of tensor.
2:08:50
Oh, let's do data type first. We'll follow that order.
2:08:56
Data type of tensor. And we're going to go, how do we do this? Some tensor dot what? Dot d type. Beautiful. And then we're going to print tensors not in the right shape. So let's go
2:09:13
shape of tensor equals some tensor dot shape. Oh, I went a bit too fast, but we could also use size. Let's just confirm that actually. We'll code that out together. From my experience, some tensor dot size, and some tensor dot shape result in the same thing. Is that true? Oh, function. Oh, that's what it is. Some tensor dot size is a function, not an attribute.
2:09:49
There we go. Which one should you use? For me, I'm probably more used to using shape. You may come across dot size as well, but just realize that they do quite the same thing except one's a function and one's an attribute. An attribute is written dot shape without the curly brackets. A function or a method is with the brackets at the end. So that's the difference between these are attributes here. D type size. We're going to change this to shape. Tensor attributes. This is what we're
2:10:18
getting. I should probably write that down. This is tensor attributes. That's the formal name for these things. And then finally, what else do we want? Tensors, what device are we looking for? Let's get rid of this, get rid of this. And then print f device tensor is on. By default, our tensor is on the CPU. So some tensor dot device. There we go. So now we've got our tensor here, some tensor. The data type is a torch float 32 because we didn't change it to anything else.
2:10:57
And torch float 32 is the default. The shape is three four, which makes a lot of sense because we passed in three four here. And the device tensor is on is the CPU, which is, of course, the default, unless we explicitly say to put it on another device, all of the tensors that we create will default to being on the CPU, rather than the GPU. And we'll see later on how to put tensors and other things in torch onto a GPU. But with that being said, give it a shot,
2:11:25
create your own tensor, get some information from that tensor, and see if you can change these around. So see if you could create a random tensor, but instead of float 32, it's a float 16. And then probably another extracurricular, we haven't covered this yet. But see how to change the device a pytorch tensor is on. Give that a crack. And I'll see you in the next video.
2:11:49
Welcome back. So in the last video, we had a look at a few tensor attributes, namely the data type of a tensor, the shape of a tensor, and the device that a tensor lives on. And I alluded to the fact that these will help resolve three of the most common issues in building neural networks, deep learning models, specifically with pytorch. So tensor has not been the right data type, tensor has not been the right shape, and tensor has not been on the right device. So now let's
2:12:19
get into manipulating tensors. And what I mean by that, so let's just write here the title, manipulating tensors. And this is going to be tensor operations. So when we're building neural networks, neural networks are comprised of lots of mathematical functions that pytorch code is going to run behind the scenes for us. So let's go here, tensor operations include addition, subtraction, and these are the regular addition, subtraction, multiplication. There's two types
2:13:02
of multiplication in that you'll typically see referenced in deep learning and neural networks, division, and matrix multiplication. And these, the ones here, so addition, subtraction, multiplication, division, your typical operations that you're probably familiar with matrix multiplication. The only different one here is matrix multiplication. We're going to have a look at that in a minute. But to find patterns in numbers of a data set, a neural network will combine these functions
2:13:37
in some way, shape or form. So it takes a tensor full of random numbers, performs some kind of combination of addition, subtraction, multiplication, division, matrix multiplication. It doesn't have to be all of these. It could be any combination of these to manipulate these numbers in some way to represent a data set. So that's how a neural network learns is it will just comprise these functions, look at some data to adjust the numbers of a random tensor, and then go from there. But
2:14:05
with that being said, let's look at a few of these. So we'll begin with addition. First thing we need to do is create a tensor. And to add something to a tensor, we'll just go torch tensor. Let's go one, two, three, add something to a tensor is tensor plus, we can use plus as the addition operator, just like in Python, tensor plus 10 is going to be tensor 11, 12, 13, tensor plus 100 is going to be as you'd expect plus 100. Let's leave that as plus 10 and add 10 to it. And so you might be
2:14:49
able to guess how we would multiply it by 10. So let's go multiply tensor by 10. We can go tensor, star, which are my keyboard shift eight, 10. We get 10, 10, 10. And because we didn't reassign it,
2:15:10
our tensor is still 123. So if we go, if we reassign it here, tensor equals tensor by 10, and then check out tensor, we've now got 10 2030. And the same thing here, we'll have 10 2030. But then if we go back from the top, if we delete this reassignment, oh, what do we get there, tensor by 10. Oh, what's happened here? Oh, because we've got, yeah, okay, I see, tensor by 10, tensor, still 123. What should we try now? How about subtract subtract 10 equals tensor minus 10.
2:15:58
And you can also use, well, there we go, one minus 10, eight minus 10, three minus 10. You can also use like torch has inbuilt functions or pytorch. So try out pytorch inbuilt functions. So torch dot mall is short for multiply. We can pass in our tensor here, and we can add in 10. That's going to multiply each element of tensor by 10. So just taking the original tensor that we created, which is 123. And performing the same thing as this,
2:16:35
I would recommend where you can use the operators from Python. If for some reason, you see torch dot mall, maybe there's a reason for that. But generally, these are more understandable if you just use the operators, if you need to do a straight up multiplication, straight up addition, or straight up subtraction, because torch also has torch dot add, torch dot add, is it torch dot add? It might be torch dot add. I'm not sure. Oh, there we go. Yeah, torch dot add. So as I alluded to before,
2:17:07
there's two different types of multiplication that you'll hear about element wise and matrix multiplication. We're going to cover matrix multiplication in the next video. As a challenge, though, I would like you to search what is matrix multiplication. And I think the first website that comes up, matrix multiplication, Wikipedia, yeah, math is fun. It has a great guide. So before we get into matrix multiplication, jump into math is fun to have a look at matrix multiplying,
2:17:38
and have a think about how we might be able to replicate that in pie torch. Even if you're not sure, just have a think about it. I'll see you in the next video. Welcome back. In the last video, we discussed some basic tensor operations, such as addition, subtraction, multiplication, element wise, division, and matrix multiplication. But we didn't actually go through what matrix multiplication is. So now let's start on that more particularly discussing the difference between
2:18:09
element wise and matrix multiplication. So we'll come down here, let's write another heading, matrix multiplication. So there's two ways, or two main ways. Yeah, let's write that two main ways of performing multiplication in neural networks and deep learning. So one is the simple version, which is what we've seen, which is element wise multiplication. And number two is matrix multiplication. So matrix multiplication is actually possibly the most common tensor operation you
2:18:49
will find inside neural networks. And in the last video, I issued the extra curriculum of having a look at the math is fun dot com page for how to multiply matrices. So the first example they go through is element wise multiplication, which just means multiplying each element by a specific number. In this case, we have two times four equals eight, two times zero equals zero, two times one equals two, two times negative nine equals negative 18. But then if we move on to matrix
2:19:23
multiplication, which is multiplying a matrix by another matrix, we need to do the dot product. So that's something that you'll also hear matrix multiplication referred to as the dot product. So these two are used interchangeably matrix multiplication or dot product. And if we just look up the symbol for dot product, you'll find that it's just a dot. There we go, a heavy dot, images. There we go, a dot B. So this is vector a dot product B. A few different options there,
2:20:00
but let's look at what it looks like in pytorch code. But first, there's a little bit of a difference here. So how did we get from multiplying this matrix here of one, two, three, four, five, six, times seven, eight, nine, 10, 11, 12? How did we get 58 there? Well, we start by going, this is the difference between element wise and dot product, by the way, one times seven. We'll record that down there. So that's seven. And then two times nine. So this is first row,
2:20:29
first column, two times nine is 18. And then three times 11 is 33. And if we add those up, seven plus 18, plus 33, we get 58. And then if we were to do that for each other element that's throughout these two matrices, we end up with something like this. So that's what I'd encourage you to go through step by step and reproduce this a good challenge would be to reproduce this by hand with pytorch code. But now let's go back and write some pytorch code to do both of these. So
2:21:04
I just want to link here as well, more information on multiplying matrices. So I'm going to turn this into markdown. Let's first see element wise, element wise multiplication. We're going to start with just a rudimentary example. So if we have our tensor, what is it at the moment? It's 123. And then if we multiply that by itself, we get 149. But let's print something out so it looks a bit prettier than that. So print, I'm going to turn this into a string. And then we do that. So if we
2:21:42
print tensor times tensor, element wise multiplication is going to give us print equals. And then let's do in here tensor times tensor. We go like that. Wonderful. So we get one times one equals one, two times two equals four, three times three equals nine. Now for matrix multiplication, pytorch stores matrix multiplication, similar to torch dot mall in the torch dot mat mall space, which stands for matrix multiplication. So let's just test it out. Let's just true the exact
2:22:25
same thing that we did here, instead of element wise, we'll do matrix multiplication on our 123 tensor. What happens here? Oh my goodness, 14. Now why did we get 14 instead of 149? Can you guess how we got to 14 or think about how we got to 14 from these numbers? So if we recall back, we saw that for we're only multiplying two smaller tensors, by the way, 123. This example is with a larger one, but the same principle applies across different sizes of tensors or matrices.
2:23:04
And when I say matrix multiplication, you can also do matrix multiplication between tensors. And in our case, we're using vectors just to add to the confusion. But what is the difference here between element wise and dot product? Well, we've got one main addition. And that is addition. So if we were to code this out by hand, matrix multiplication by hand, we'd have recall that the elements of our tensor are 123. So if we wanted to matrix multiply that by itself,
2:23:40
we'd have one times one, which is the equivalent of doing one times seven in this visual example. And then we'd have plus, it's going to be two times two, two times two. What does that give us? Plus three times three. What does it give us? Three times three. That gives us 14. So that's how we got to that number there. Now we could do this with a for loop. So let's have a gaze at when I say gaze, it means have a look. That's a Australian colloquialism for having a look. But I want to
2:24:20
show you the time difference in it might not actually be that big a difference if we do it by hand versus using something like matmore. And that's another thing to note is that if PyTorch has a method already implemented, chances are it's a fast calculating version of that method. So I know for basic operators, I said it's usually best to just use this straight up basic operator. But for something like matrix multiplication or other advanced operators instead of the basic
2:24:50
operators, you probably want to use the torch version rather than writing a for loop, which is what we're about to do. So let's go value equals zero. This is matrix multiplication by hand. So for I in range, len tensor, so for each element in the length of our tensor, which is 123, we want to update our value to be plus equal, which is doing this plus reassignment here. The ith element in each tensor times the ith element. So times itself. And then how long is this going to take?
2:25:29
Let's now return the value. We should get 14, print 14. There we go. So 1.9 milliseconds on whatever CPU that Google collab is using behind the scenes. But now if we time it and use the torch method torch dot matmore, it was tensor dot sensor. And again, we're using a very small tensor. So okay, there we go. It actually showed how much quicker it is, even with such a small tensor. So this is 1.9 milliseconds. This is 252 microseconds. So this is 10 times slower using a for loop,
2:26:13
then pie torches vectorized version. I'll let you look into that if you want to find out what vectorization means. It's just a type of programming that rather than writing for loops, because as you could imagine, if this tensor was, let's say, had a million elements instead of just three, if you have to loop through each of those elements one by one, that's going to be quite cumbersome. So a lot of pie torches functions behind the scenes implement optimized functions to perform
2:26:45
mathematical operations, such as matrix multiplication, like the one we did by hand, in a far faster manner, as we can see here. And that's only with a tensor of three elements. So you can imagine the speedups on something like a tensor with a million elements. But with that being said, that is the crux of matrix multiplication. For a little bit more, I encourage you to read through this documentation here by mathisfun.com. Otherwise,
2:27:12
let's look at a couple of rules that we have to satisfy for larger versions of matrix multiplication. Because right now, we've done it with a simple tensor, only 123. Let's step things up a notch in the next video. Welcome back. In the last video, we were introduced to matrix multiplication, which although we haven't seen it yet, is one of the most common operations in neural networks. And we saw that you should always try to use torches implementation of certain operations,
2:27:46
except if they're basic operations, like plus multiplication and whatnot, because chances are it's a lot faster version than if you would do things by hand. And also, it's a lot less code. Like compared to this, this is pretty verbose code compared to just a matrix multiply these two tensors. But there's something that we didn't allude to in the last video. There's a couple of rules that need to be satisfied when performing matrix multiplication.
2:28:14
It worked for us because we have a rather simple tensor. But once you start to build larger tensors, you might run into one of the most common errors in deep learning. I'm going to write this down actually here. This is one to be very familiar with. One of the most common errors in deep learning, we've already alluded to this as well, is shape errors. So let's jump back to this in a minute. I just want to write up here. So there are two rules that performing or two main rules
2:28:50
that performing matrix multiplication needs to satisfy. Otherwise, we're going to get an error. So number one is the inner dimensions must match. Let's see what this means. So if we want to have two tensors of shape, three by two, and then we're going to use the at symbol.
2:29:15
Now, we might be asking why the at symbol. Well, the at symbol is another, is a like an operator symbol for matrix multiplication. So I just want to give you an example. If we go tensor at
2:29:29
at stands for matrix multiplication, we get tensor 14, which is exactly the same as what we got there. Should you use at or should you use mat mall? I would personally recommend to use mat mall. It's a little bit clearer at sometimes can get confusing because it's not as common as seeing something like mat mall. So we'll get rid of that, but I'm just using it up here for brevity. And then we're going to go three, two. Now, this won't work. We'll see why in a second.
2:29:56
But if we go two, three, at, and then we have three, two, this will work. Or, and then if we go the reverse, say threes on the outside, twos here. And then we have twos on the inside and threes on the outside, this will work. Now, why is this? Well, this is the rule number one. The inner dimensions must match. So the inner dimensions are what I mean by this is let's create torch round or create of size 32. And then we'll get its shape. So we have, so if we created a tensor
2:30:36
like this, three, two, and then if we created another tensor, well, let me just show you straight up torch dot mat mall torch dot ran to watch this won't work. We'll get an error. There we go. So this is one of the most common errors that you're going to face in deep learning is that matrix one and matrix two shapes cannot be multiplied because it doesn't satisfy rule number one. The inner dimensions must match. And so what I mean by inner dimensions is this dimension multiplied
2:31:11
by this dimension. So say we were trying to multiply three, two by three, two, these are the inner dimensions. Now this will work because why the inner dimensions match. Two, three by three, two, two, three by three, two. Now notice how the inner dimensions, inner, inner match. Let's see what comes out here. Look at that. And now this is where rule two comes into play. Two. The resulting matrix has the shape of the outer dimensions. So we've just seen this one two, three at three, two,
2:32:04
which is at remember is matrix multiply. So we have a matrix of shape, two, three, matrix multiply a matrix of three, two, the inner dimensions match. So it works. The resulting shape is what? Two, two. Just as we've seen here, we've got a shape of two, two. Now what if we did the reverse? What if we did this one that also will work? Three on the outside. What do you think is going to happen here? In fact, I encourage you to pause the video and give it a go. So this
2:32:44
is going to result in a three three matrix. But don't take my word for it. Let's have a look. Three, put two on the inside and we'll put two on the inside here and then three on the outside. What does it give us? Oh, look at that. A three three. One, two, three. One, two, three. Now what if we were to change this? Two and two. This can be almost any number you want. Let's change them both to 10. What's going to happen? Will this work? What's the resulting shape going to be? So the
2:33:18
inner dimensions match? What's rule number two? The resulting matrix has the shape of the outer dimension. So what do you think is going to be the shape of this resulting matrix multiplication? Well, let's have a look. It's still three three. Wow. Now what if we go 10? 10 on the outside
2:33:40
and 10 and 10 on the inside? What do we get? Well, we get, I'm not going to count all of those, but if we just go shape, we get 10 by 10. Because these are the two main rules of matrix multiplication is if you're running into an error that the matrix multiplication can't work. So let's say this was 10 and this was seven. Watch what's going to happen? We can't multiply them because the inner dimensions do not match. We don't have 10 and 10. We have 10 and seven. But then when we change
2:34:10
this so that they match, we get 10 and 10. Beautiful. So now let's create a little bit more of a specific example. We'll create two tenses. We'll come down. Actually, to prevent this video from being too long, I've got an error in the word error. That's funny. We'll go on with one of those common errors in deep learning shape errors. We've just seen it, but I'm going to get a little bit more specific with that shape error in the next video. Before we do that, have a look at matrix
2:34:38
multiplication. There's a website, my other favorite website. I told you I've got two. This is my other one. Matrix multiplication dot XYZ. This is your challenge before the next video. Put in some random numbers here, whatever you want, two, 10, five, six, seven, eight, whatever you want. Change these around a bit, three, four. Well, that's a five, not a four. And then multiply and just watch what happens. That's all I'd like you to do. Just watch what happens and we're going to replicate
2:35:10
something like this in PyTorch code in the next video. I'll see you there. Welcome back. In the last video, we discussed a little bit more about matrix multiplication, but we're not done there. We looked at two of the main rules of matrix multiplication, and we saw a few errors of what happens if those rules aren't satisfied, particularly if the inner dimensions don't match. So this is what I've been alluding to as one of the most common
2:35:41
errors in deep learning, and that is shape errors. Because neural networks are comprised of lots of matrix multiplication operations, if you have some sort of tensor shape error somewhere in your neural network, chances are you're going to get a shape error. So now let's investigate how we can deal with those. So let's create some tenses, shapes for matrix multiplication. And I also showed you the website, sorry, matrix multiplication dot xyz. I hope you had a go at
2:36:12
typing in some numbers here and visualizing what happens, because we're going to reproduce something very similar to what happens here, but with PyTorch code. Shapes for matrix multiplication, we have tensor a, let's create this as torch dot tensor. We're going to create a tensor with just the elements one, two, all the way up to, let's just go to six, hey, that'll be enough. Six, wonderful. And then tensor b can be equal to a torch tensor
2:36:47
of where we're going to go for this one. Let's go seven, 10, this will be a little bit confusing this one, but then we'll go eight, 11, and this will go up to 12, nine, 12. So it's the same
2:37:04
sort of sequence as what's going on here, but they've been swapped around. So we've got the vertical axis here, instead of one, two, three, four, this is just seven, eight, nine, 10, 11, 12. But let's now try and perform a matrix multiplication. How do we do that? Torch dot mat mall for matrix multiplication. PS torch also has torch dot mm, which stands for matrix multiplication, which is a short version. So I'll just write down here so that you know
2:37:30
tensor a, tensor b. I'm going to write torch dot mm is the same as torch dot mat mall. It's an alias for writing less code. This is literally how common matrix multiplications are in PyTorch is that they've made torch dot mm as an alias for mat mall. So you have to type four less characters using torch dot mm instead of mat mall. But I like to write mat mall because it's a little bit like it explains what it does a little bit more than mm. So what do you think's going to happen
2:38:09
here? It's okay if you're not sure. But what you could probably do to find out is check the shapes of these. Does this operation matrix multiplication satisfy the rules that we just discussed? Especially this one. This is the main one. The inner dimensions must match. Well, let's have a look, hey? Oh, no, mat one and mat two shapes cannot be multiplied. Three by two and three by two. This is very similar to what we went through in the last video.
2:38:39
But now we've got some actual numbers there. Let's check the shape.
2:38:44
Oh, torch size three two. Torch size three two now. In the last video we created a random tensor and we could adjust the shape on the fly. But these tensors already exist. How might we adjust the shape of these? Well, now I'm going to introduce you to another very common operation or tensor manipulation that you'll see. And that is the transpose. To fix our tensor shape issues, we can manipulate the shape of one of our tensors using a transpose. And so, all right here,
2:39:32
we're going to see this anyway, but I'm going to define it in words. A transpose switches the axes or dimensions of a given tensor. So let's see this in action. If we go, and the way to do it, is you can go tensor b dot t. Let's see what happens. Let's look at the original tensor b as well. So dot t stands for transpose. And that's a little bit hard to read, so we might do these on different lines, tensor b. We'll get rid of that. So you see what's happened here. Instead of
2:40:18
tensor b, this is the original one. We might put the original on top. Instead of the original one having seven, eight, nine, 10, 11, 12 down the vertical, the transpose has transposed it to seven, eight, nine across the horizontal and 10, 11, 12 down here. Now, if we get the shape of this, tensor b dot shape, let's have a look at that. Let's have a look at the original shape, tensor b dot shape. What's happened? Oh, no, we've still got three, two. Oh, that's what I've missed out here.
2:40:55
I've got a typo. Excuse me. I thought I was, you think code that you've written is working, but then you realize you've got something as small as just a dot t missing, and it throws off your whole train of thought. So you're seeing these arrows on the fly here. Now, tensor b is this, but its shape is torch dot size three, two. And if we try to matrix multiply three, two, and three, two, tensor a and tensor b, we get an error. Why? Because the inner dimensions do not match.
2:41:26
But if we perform a transpose on tensor b, we switch the dimensions around. So now, we perform a transpose with tensor b dot t, t's for transpose. We have, this is the important point as well. We still have the same elements. It's just that they've been rearranged. They've been transposed. So now, tensor b still has the same information encoded, but rearranged. So now we have torch size two, three. And so when we try to matrix multiply these,
2:42:02
we satisfy the first criteria. And now look at the output of the matrix multiplication of tensor a and tensor b dot t transposed is three, three. And that is because of the second rule of matrix multiplication. The resulting matrix has the shape of the outer dimensions. So we've got three, two matrix multiply two, three results in a shape of three, three. So let's predify some of this, and we'll print out what's going on here. Just so we know, we can step through it,
2:42:36
because right now we've just got codal over the place a bit. Let's see here, the matrix multiplication operation works when tensor b is transposed. And in a second, I'm going to show you what this looks like visually. But right now we've done it with pytorch code, which might be a little confusing. And that's perfectly fine. Matrix multiplication takes a little while and a little practice. So original shapes is going to be tensor a dot shape. Let's
2:43:10
see what this is. And tensor b equals tensor b dot shape. But the reason why we're spending so much time on this is because as you'll see, as you get more and more into neural networks and deep learning, the matrix multiplication operation is one of the most or if not the most common. Same shape as above, because we haven't changed tensor a shape, we've only changed tensor b shape, or we've transposed it. And then in tensor b dot transpose equals, we want tensor b dot
2:43:50
t dot shape. Wonderful. And then if we print, let's just print out, oops, print, I spelled the wrong word there, print. We want, what are we multiplying here? This is one of the ways, remember our motto of visualize, visualize, visualize, well, this is how I visualize, visualize, visualize things, shape, let's do the at symbol for brevity, tensor, and let's get b dot t dot shape. We'll put down our little rule here, inner dimensions must match. And then print,
2:44:29
let's get the output output, I'll put that on a new line. The output is going to equal torch dot, or our outputs already here, but we're going to rewrite it for a little bit of practice, tensor a, tensor b dot t. And then we can go print output. And then finally, print, let's get it on a new line as well, the output shape, a fair bit going on here. But we're going to step through it, and it's going to help us understand a little bit about what's going on. That's the data visualizes
2:45:06
motto. There we go. Okay, so the original shapes are what torch size three two, and torch size three two, the new shapes tensor a stays the same, we haven't changed tensor a, and then we have tensor b dot t is torch size two three, then we multiply a three by two by a two by three. So the inner dimensions must match, which is correct, they do match two and two. Then we have an output of tensor at 27, 30, 33, 61, 68, 75, etc. And the output shape is what the output shape is the outer
2:45:45
dimensions three three. Now, of course, you could rearrange this maybe transpose tensor a instead of tensor b, have a play around with it. See if you can create some more errors trying to multiply these two, and see what happens if you transpose tensor a instead of tensor b, that's my challenge. But before we finish this video, how about we just recreate what we've done here with this cool website matrix multiplication. So what did we have? We had tensor a, which is one to six, let's recreate
2:46:17
this, remove that, this is going to be one, two, three, four, five, six, and then we want to increase this, and this is going to be seven, eight, nine, 10, 11, 12. Is that the right way of doing things? So this is already transposed, just to let you know. So this is the equivalent of tensor b on the right here, tensor b dot t. So let me just show you, if we go tensor b dot transpose,
2:46:55
which original version was that, but we're just passing in the transpose version to our matrix multiplication website. And then if we click multiply, this is what's happening behind the scenes with our pytorch code of matmore. We have one times seven plus two times 10. Did you see that little flippy thing that it did? That's where the 27 comes from. And then if we come down here, what's our first element? 27 when we matrix multiply them. Then if we do the same thing,
2:47:22
the next step, we get 30 and 61, from a combination of these numbers, do it again, 33, 68, 95, from a combination of these numbers, again, and again, and finally we end up with exactly what we have here. So that's a little bit of practice for you to go through is to create some of your own tensors can be almost whatever you want. And then try to matrix multiply them with different shapes. See what happens when you transpose and what different values you get.
2:47:54
And if you'd like to visualize it, you could write out something like this. That really helps me understand matrix multiplication. And then if you really want to visualize it, you can go through this website and recreate your target tensors in something like this. I'm not sure how long you can go. But yeah, that should be enough to get started. So give that a try and I'll see you in the next video.
2:48:17
Welcome back. In the last few videos, we've covered one of the most fundamental operations in neural networks. And that is matrix multiplication. But now it's time to move on. And let's cover tensor aggregation. And what I mean by that is finding the min, max, mean, sum, et cetera, tensor aggregation of certain tensor values. So for whatever reason, you may want to find the minimum value of a tensor, the maximum value, the mean, the sum, what's going on
2:48:49
there. So let's have a look at some few PyTorch methods that are in built to do all of these. And again, if you're finding one of these values, it's called tensor aggregation because you're going from what's typically a large amount of numbers to a small amount of numbers. So the min of this tensor would be 27. So you're turning it from nine elements to one element, hence aggregation. So let's create a tensor, create a tensor, x equals torch dot, let's use a range.
2:49:20
We'll create maybe a zero to 100 with a step of 10. Sounds good to me. And we can find the min
2:49:30
by going, can we do torch dot min? Maybe we can. Or we could also go
2:49:38
x dot min. And then we can do the same, find the max torch dot max and x dot max. Now how do you think we might get the average? So let's try it out. Or find the mean, find the mean torch dot mean x. Oops, we don't have an x. Is this going to work? What's happened? Mean input data type should be either floating point or complex D types got long instead. Ha ha. Finally, I knew the error would show its face eventually. Remember how I said it right up here that
2:50:24
we've covered a fair bit already. But right up here, some of the most common errors that you're going to run into is tensor is not the right data type, not the right shape. We've seen that with matrix multiplication, not the right device. We haven't seen that yet. But not the right data type. This is one of those times. So it turns out that the tensor that we created, x is of the data type, x dot D type.
2:50:48
In 64, which is long. So if we go to, let's look up torch tensor. This is where they're getting long from. We've seen long before is N64. Where's that or long? Yeah. So long tenter. That's what it's saying. And it turns out that the torch mean function can't work on tensors with data type long. So what can we do here? Well, we can change the data type of x. So let's go torch mean x type and change it to float 32. Or before we do that,
2:51:25
if we go to torch dot mean, is this going to tell us that it needs a D type? Oh, D type. One option on the desired data type. Does it have float 32? It doesn't tell us. Ah, so this is another one of those little hidden things that you're going to come across. And you only really come across this by writing code is that sometimes the documentation doesn't really tell you explicitly what D type the input should be, the input tensor. However, we find out that with this error message
2:52:06
that it should either be a floating point or a complex D type, not along. So we can convert it to torch float 32. So all we've done is gone x type as type float 32. Let's see what happens here. 45 beautiful. And then the same thing, if we went, can we do x dot mean? Is that going to work as well?
2:52:29
Oh, same thing. So if we go x dot type torch dot float 32, get the mean of that. There we go. So that is, I knew it would come up eventually. A beautiful example of finding the right data type. Let me just put a note here. Note the torch dot mean function requires a tensor of float 32. So so far, we've seen two of the major errors in PyTorch is data type and shape issues. What's another one that we said? Oh, some. So find the sum. Find the sum we want x dot sum or maybe we
2:53:12
just do torch dot sum first. Keep it in line with what's going on above and x dot sum.
2:53:18
Which one of these should you use like torch dot something x or x dot sum? Personally, I prefer torch dot max, but you'll also probably see me at points right this. It really depends on what's going on. I would say pick whichever style you prefer. And because behind the scenes, they're calling the same methodology. Picture whichever style you prefer and stick with that throughout your code. For now, let's leave it at that tensor aggregation. There's some
2:53:47
finding min max mean sum. In the next video, we're going to look at finding the positional min and max, which is also known as arg max and arg min or vice versa. So actually, that's a little bit of a challenge for the next video is see how you can find out what the positional min and max is of this. And what I mean by that is which index does the max value occur at and which index of this tensor does the min occur at? You'll probably want to look into the methods
2:54:17
arg min torch dot arg min for that one and torch dot arg max for that. But we'll cover that in the next video. I'll see you there. Welcome back. In the last video, we learned all about tensor aggregation. And we found the min the max the mean and the sum. And we also ran into one of the most common issues in pie torch and deep learning and neural networks in general. And that was wrong data types. And so we solved that issue by converting because some functions such as torch dot mean
2:54:50
require a specific type of data type as input. And we created our tensor here, which was of by default torch in 64. However, torch dot mean requires torch dot float 32. We saw that in an error. We fix that by changing the type of the inputs. I also issued you the challenge of finding
2:55:10
finding the positional min and max. And you might have found that you can use the arg min for the minimum. Let's remind ourselves of what x is x. So this means at tensor index of tensor x. If we find the argument, that is the minimum value, which is zero. So at index zero, we get the value zero. So that's at zero there. Zero there. This is an index value. So this is what arg min stands for find the position in tensor that has the minimum value with arg min.
2:55:57
And then returns index position of target tensor where the minimum value occurs. Now, let's just change x to start from one,
2:56:18
just so there we go. So the arg min is still position zero, position zero. So this is an index value. And then if we index on x at the zeroth index, we get one. So the minimum value in x is one. And then the maximum, you might guess, is find the position in tensor that has the maximum value with arg max. And it's going to be the same thing, except it'll be the maximum, which is, which position index nine. So if we go zero, one, two, three, four, five, six, seven, eight,
2:57:00
nine. And then if we index on x for the ninth element, we get 91 beautiful. Now these two are useful for if yes, you want to define the minimum of a tensor, you can just use min. But if you sometimes you don't want the actual minimum value, you just want to know where it appears, particularly with the arg max value. This is helpful for when we use the soft max activation function later on. Now we haven't covered that yet. So I'm not going to allude too much to it.
2:57:32
But just remember to find the positional min and max, you can use arg min and arg max. So that's all we need to cover with that. Let's keep going in the next video. I'll see you then.
2:57:47
Welcome back. So we've covered a fair bit of ground. And just to let you know, I took a little break after going through all of these. And I'd just like to show you how I get back to where I'm at, because if we tried to just write x here and press shift and enter, because our collab was disconnected, it's now connecting because as soon as you press any button in collab, it's going to reconnect. It's going to try to connect, initialize, and then x is probably not going to
2:58:15
be stored in memory anymore. So there we go. Name x is not defined. That's because the collab state gets reset if you take a break for a couple of hours. This is to ensure Google can keep providing resources for free. And it deletes everything to ensure that there's no compute resources that are being wasted. So to get back to here, I'm just going to go restart and run all. You don't necessarily have to restart the notebook. You could also go, do we have run all? Yeah,
2:58:44
we could do run before. That'll run every cell before this. We could run after we could run the selection, which is this cell here. I'm going to click run all, which is just going to go through every single cell that we've coded above and run them all. However, it will also stop at the errors where I've left in on purpose. So remember when we ran into a shape error? Well, because this error, we didn't fix it. I left it there on purpose so that we could keep seeing a shape error. It's
2:59:11
going to stop at this cell. So we're going to have to run every cell after the error cell. So see how it's going to run these now. They run fine. And then we get right back to where we were, which was X. So that's just a little tidbit of how I get back into coding. Let's now cover reshaping,
2:59:32
stacking, squeezing, and unsqueezing. You might be thinking, squeezing and unsqueezing. What are you talking about, Daniel? Well, it's all to do with tenses. And you're like, are we going to squeeze our tenses? Give them a hug. Are we going to let them go by unsqueezing them? Well, let's quickly define what these are. So reshaping is we saw before one of the most common errors in machine learning and deep learning is shape mismatches with matrices because they
3:00:01
have to satisfy certain rules. So reshape reshapes an input tensor to a defined shape. Now, we're just defining these things in words right now, but we're going to see it in code in just a minute. There's also view, which is return a view of an input tensor of certain shape, but keep the same memory as the original tensor. So we'll see what view is in a second. Reshaping and view are quite similar, but a view always shares the same memory as the original
3:00:40
tensor. It just shows you the same tensor, but from a different perspective, a different shape. And then we have stacking, which is combine multiple tensors on top of each other. This is a V stack for vertical stack or side by side. H stack. Let's see what different types of torch stacks there are. Again, this is how I research different things. If I wanted to learn something new, I would search torch something stack concatenate a sequence of tensors along a new dimension. Okay. So maybe we
3:01:16
not H stack or V stack, we can just define what dimension we'd like to combine them on. I wonder if there is a torch V stack. Torch V stack. Oh, there it is. And is there a torch H stack for horizontal stack? There is a H stack. Beautiful. So we'll focus on just the plain stack. If you want to have a look at V stack, it'll be quite similar to what we're going to do with stack and same with H stack. Again, this is just words for now. We're going to see the code in a minute.
3:01:43
So there's also squeeze, which removes all one dimensions. I'm going to put one in code, dimensions from a tensor. We'll see what that looks like. And then there's unsqueeze, which adds a one dimension to our target tensor. And then finally, there's permute, which is return a view of the input with dimensions permuted. So swapped in a certain way. So a fair few methods here. But essentially the crust of all of these, the main point of all of these is to manipulate
3:02:32
our tensors in some way to change their shape or change their dimension. Because again, one of the number one issues in machine learning and deep learning is tensor shape issues. So let's start off by creating a tensor and have a look at each of these. Let's create a tensor. And then we're going to just import torch. We don't have to, but this will just enable us to run the notebook directly from this cell if we wanted to, instead of having to run everything above here. So let's
3:03:02
create another X torch dot a range because range is deprecated. I'm just going to add a few code cells here so that I can scroll and that's in the middle of the screen there. Beautiful. So let's just make it between one and 10 nice and simple. And then let's have a look at X and X dot shape. What does this give us? Okay, beautiful. So we've got the numbers from one to nine. Our tensor is of shape torch size nine. Let's start with reshape. So how about we add an extra dimension. So then
3:03:40
we have X reshaped equals X dot reshape. Now a key thing to keep in mind about the reshape is that the dimensions have to be compatible with the original dimensions. So we're going to change the shape of our original tensor with a reshape. And we try to change it into the shape one seven. Does that work with the number nine? Well, let's find out, hey, let's check out X reshaped. And then we'll look at X reshaped dot shape. What's this going to do? Oh, why do we get an error there?
3:04:16
Well, it's telling us here, this is what pie torch is actually really good at is giving us errors for what's going wrong. We have one seven is invalid for input size of nine. Well, why is that? Well, we're trying to squeeze nine elements into a tensor of one times seven into seven elements. But if we change this to nine, what do we get? Ah, so do you notice what just happened here? We just added a single dimension. See the single square bracket with
3:04:45
the extra shape here. What if we wanted to add two? Can we do that? No, we can't. Why is that? Well, because two nine is invalid for input size nine, because two times nine is what? 18. So we're trying to double the amount of elements without having double the amount of elements. So if we change this back to one, what happens if we change these around nine one? What does this do? Oh, a little bit different there. So now instead of adding one on the first dimension or
3:05:14
the zeroth dimension, because Python is zero indexed, we added it on the first dimension, which is giving us a square bracket here if we go back. So we add it to the outside here, because we've put the one there. And then if we wanted to add it on the inside,
3:05:32
we put the one on the outside there. So then we've got the torch size nine one. Now, let's try change the view, change the view. So just to reiterate, the reshape has to be compatible with the original size. So how about we change this to one to 10? So we have a size of 10, and then we can go five, two, what happens there? Oh, it's compatible because five times two equals 10. And then what's another way we could do this? How about we make it up to 12? So we've got 12
3:06:05
elements, and then we can go three, four, a code cells taking a little while run here. Then we'll go back to nine, just so we've got the original there.
3:06:22
Whoops, they're going to be incompatible. Oh, so this is another thing. This is good. We're getting some errors on the fly here. Sometimes you'll get saved failed with Google CoLab, and automatic saving failed. What you can do to fix this is just either keep coding, keep running some cells, and CoLab will fix itself in the background, or restart the notebook, close it, and open again. So we've got size nine, or size eight, sorry, incompatible.
3:06:54
But this is good. You're seeing the errors that come up on the fly, rather than me sort of just telling you what the errors are, you're seeing them as they come up for me. I'm trying to live code this, and this is what's going to happen when you start to use Google CoLab, and subsequently other forms of Jupyter Notebooks. But now let's get into the view, so we can go z equals, let's change the view of x. View will change it to one nine, and then we'll go z, and then z dot shape.
3:07:29
Ah, we get the same thing here. So view is quite similar to reshape. Remember, though, that a view shares the memory with the original tensor. So z is just a different view of x. So z shares the same memory as what x does. So let's exemplify this. So changing z changes x, because a view of a tensor shares the same memory as the original input. So let's just change z, change the first element by using indexing here. So we're targeting one, we'll set this to equal five, and then we'll
3:08:11
see what z and x equal. Yeah, so see, we've got z, the first one here, we change the first element, the zero element to five. And the same thing happens with x, we change the first element of z. So because z is a view of x, the first element of x changes as well. But let's keep going. How about we stack some tenses on top of each other? And we'll see what the stack function does in torch. So stack tenses on top of each other. And I'll just see if I press command S to save,
3:08:48
maybe we'll get this fixed. Or maybe it just will fix itself. Oh, notebook is saved.
3:08:56
Unless you've made some extensive changes that you're worried about losing, you could just download this notebook, so file download, and upload it to collab. But usually if you click yes,
3:09:08
it sort of resolves itself. Yeah, there we go. All changes saved. So that's beautiful troubleshooting on the fly. I like that. So x stack, let's stack some tenses together, equals torch stack. Let's go x x x, because if we look at what the doc string of stack is, will we get this in collab? Or we just go to the documentations? Yeah. So list, it takes a list of tenses and concatenates a sequence of tenses along a new dimension. And we define the dimension,
3:09:37
the dimension by default is zero. That's a little bit hard to read for me. So tenses, dim equals zero. If we come into here, the default dimension is zero. Let's see what happens when we play around with the dimension here. So we've got four x's. And the first one, we'll just do it by default, x stack. Okay, wonderful. So they're stacked vertically. Let's see what happens if we change this to one. Oh, they rearranged a little and stack like that. What happens if we change it
3:10:07
to two? Does it have a dimension to? Oh, we can't do that. Well, that's because the original shape of x is incompatible with using dimension two. So the only real way to get used to what happens here by stacking them on top of each other is to play around with the different values for the dimension. So dim zero, dim one, they look a little bit different there. Now they're on top of each other. And so the first zero index is now the zeroth tensor. And then same with two being there,
3:10:37
three and so on. But we'll leave it at the default. And there's also v stack and h stack. I'll leave that to you to to practice those. But I think from memory v stack is using dimension equals zero. Or h stack is like using dimension equals one. I may have those back the front. You can correct me if I'm wrong there. Now let's move on. We're going to now have a look at squeeze and unsqueeze. So actually, I'm going to get you to practice this. So see if you can look up torch squeeze
3:11:10
and torch unsqueeze. And see if you can try them out. We've created a tensor here. We've used reshape and view and we've used stack. The usage of squeeze and unsqueeze is quite similar. So give that a go. And to prevent this video from getting too long, we'll do them together in the next video.
3:11:29
Welcome back. In the last video, I issued the challenge of trying out torch dot squeeze, which removes all single dimensions from a target tensor. And how would you try that out? Well, here's what I would have done. I'd go to torch dot squeeze and see what happens. Open up the documentation. Squeeze input dimension returns a tensor with all the dimensions of input size one removed. And does it have some demonstrations? Yes, it does. Wow. Okay. So you could copy this in
3:12:04
straight into a notebook, copy it here. But what I'd actually encourage you to do quite often is if you're looking up a new torch method you haven't used, code all of the example by hand. And then just practice what the inputs and outputs look like. So x is the input here. Check the size of x, squeeze x, well, set the squeeze of x to y, check the size of y. So let's replicate something similar to this. We'll go into here, we'll look at x reshaped and we'll remind ourselves of x reshaped
3:12:38
dot shape. And then how about we see what x reshaped dot squeeze looks like. Okay. What happened here? Well, we started with two square brackets. And we started with a shape of one nine and removes all single dimensions from a target tensor. And now if we call the squeeze method on x reshaped, we only have one square bracket here. So what do you think the shape of x reshaped dot squeeze is going to be? We'll check the shape here. It's just nine. So that's the squeeze method,
3:13:17
removes all single dimensions. If we had one one nine, it would remove all of the ones. So it would just end up being nine as well. Now, let's write some print statements so we can have a little pretty output. So previous tensor, this is what I like to do. This is a form of visualize, visualize, visualize. If I'm trying to get my head around something, I print out each successive change to see what's happening. That way, I can go, Oh, okay. So that's what it was there. And then I
3:13:51
called that line of code there. Yes, it's a bit tedious. But you do this half a dozen times, a fair few times. I mean, I still do it a lot of the time, even though I've written thousands of lines of machine learning code. But it starts to become instinct after a while, you start to go, Oh, okay, I've got a dimension mismatch on my tensors. So I need to squeeze them before I put them into a certain function. For a little while, but with practice, just like riding a bike, right? But that
3:14:23
try saying is like when you first start, you're all wobbly all over the place having to look up the documentation, not that there's much documentation for riding a bike, you just kind of keep trying. But that's the style of coding. I'd like you to adopt is to just try it first. Then if you're stuck, go to the documentation, look something up, print it out like this, what we're doing, quite cumbersome. But this is going to give us a good explanation for what's happening. Here's our
3:14:47
previous tensor x reshaped. And then if we look at the shape of x reshaped, it's one nine. And then if we call the squeeze method, which removes all single dimensions from a target tensor, we have the new tensor, which is has one square bracket removed. And the new shape is all single dimensions removed. So it's still the original values, but just a different dimension. Now, let's do the same as what we've done here with unsqueeze. So we've given our tensors a hug and
3:15:14
squeezed out all the single dimensions of them. Now we're going to unsqueeze them. We're going to take a step back and let them grow a bit. So torch unsqueeze adds a single dimension
3:15:26
to a target tensor at a specific dim dimension. Now that's another thing to note in PyTorch whenever it says dim, that's dimension as in this is a zeroth dimension, first dimension. And if there was more here, we'd go two, three, four, five, six, et cetera. Because why tensors can have unlimited dimensions. So let's go previous target can be excused. So we'll get this squeezed version of our tensor, which is x squeezed up here. And then we'll go print. The previous shape
3:16:02
is going to be x squeezed dot shape. And then we're going to add an extra dimension with unsqueeze.
3:16:17
There we go, x unsqueezed equals x squeezed. So our tensor before that we remove the single dimension. And we're going to put in unsqueeze, dim, we'll do it on the zeroth dimension. And I want you to have a think about what this is going to output even before we run the code. Just think about, because we've added an extra dimension on the zeroth dimension, what's the new shape of the unsqueeze tensor going to be? So we're going to go x unsqueezed.
3:16:47
And then we're going to go print, we'll get our new tensor shape, which is going to be x unsqueezed dot shape. All right, let's have a look. There we go. So there's our previous tensor, which is the squeezed version, just as a single dimension here. And then we have our new tensor, which with the unsqueeze method on dimension zero, we've added a square bracket on the zeroth dimension, which is this one here. Now what do you think's going to happen if I change this to one?
3:17:20
Where's the single dimension going to be added? Let's have a look. Ah, so instead of adding the single dimension on the zeroth dimension, we've added it on the first dimension here. It's quite confusing because Python is zero index. So I kind of want to my brain's telling me to say first, but it's really the zeroth index here or the zeroth dimension. Now let's change this back to zero. But that's just another way of exploring things. Every time there's like a parameter that
3:17:52
we have here, dim equals something like that could be shape, could be size, whatever, try changing the values. That's what I'd encourage you to do. And even write some print code like we've done here. Now there's one more we want to try out. And that's permute. So torch dot permute rearranges the dimensions of a target tensor in a specified order. So if we wanted to check out, let's get rid of some of these extra tabs. Torch dot permute. Let's have a look. This one took me
3:18:30
a little bit of practice to get used to. Because again, working with zeroth dimensions, even though it seems like the first one. So returns a view. Okay. So we know that a view shares the memory of the original input tensor with its dimensions permuted. So permuted for me, I didn't really know what that word meant. I just have mapped in my own memory that permute means rearrange dimensions. So the example here is we start with a random tensor, we check the size, and then we'd have
3:18:58
torch permute. We're going to swap the order of the dimensions. So the second dimension is first, the zeroth dimension is in the middle, and the first dimension is here. So these are dimension values. So if we have torch random two, three, five, two, zero, one has changed this one to be over here. And then zero, one is two, three, and now two, three there. So let's try something similar to this. So one of the common places you'll be using permute, or you might see permute being
3:19:30
used is with images. So there's a data specific data format. We've kind of seen a little bit before, not too much. Original equals torch dot rand size equals. So an image tensor, we go height width color channels on the end. So I'll just write this down. So this is height width color channels. Remember, much of, and I'm going to spell color Australian style, much of deep learning is turning your data into numerical representations. And this is quite common
3:20:04
numerical representation of image data. You have a tensor dimension for the height, a tensor dimension for the width, and a tensor dimension for the color channels, which is red, green, and blue, because a certain number of red, green, and blue creates almost any color. Now, if we want to permute this, so permute the original tensor to rearrange the axis or dimension, axis or dimension, are kind of used in the same light for tensors or dim order. So let's switch the color channels
3:20:40
to be the first or the zeroth dimension. So instead of height width color channels, it'll be color channels height width. How would we do that with permute? Let's give it a shot. X permuted equals X original dot permute. And we're going to take the second dimension, because this takes a series of dims here. So the second dimension is color channels. Remember, zero, one, two. So two, we want two first, then we want the height, which is a zero. And then we
3:21:13
want the width, which is one. And now let's do this shifts, axis, zero to one, one to two, and two to zero. So this is the order as well. This two maps to zero. This zero maps to the first index. This one maps to this index. But that's enough talk about it. Let's see what it looks like. So print, previous shape, X original dot shape. And then we go here, print new shape. This will be the permuted version. We want X permuted dot shape. Let's see what this looks like. Wonderful.
3:22:01
That's exactly what we wanted. So you see, let's just write a little note here. Now this is color channels, height, width. So the same data is going to be in both of these tenses. So X original X permuted, it's just viewed from a different point of view. Because remember, a permute is a view. And what did we discuss? A view shares the same memory as the original tensor. So X permuted will share the same place in memory as X original, even though it's from a different
3:22:32
shape. So a little challenge before we move on to the next video for you, or before you move on to the next video, try change one of the values in X original. Have a look at X original. And see if that same value, it could be, let's get one of this zero, zero, get all of the dimensions here, zero. See what that is? Or can we get a single value maybe? Oops. Oh, no, we'll need a zero here, getting some practice on indexing here. Oh, zero, zero, zero. There we go. Okay, so maybe
3:23:14
we set that to some value, whatever you choose, and see if that changes in X permuted. So give that a shot, and I'll see you in the next video. Welcome back. In the last video, we covered squeezing, unsqueezing, and permuting, which I'm not going to lie, these concepts are quite a lot to take in, but just so you're aware of them. Remember, what are they working towards? They're helping us fix shape and dimension issues with our tensors, which is one of the most common
3:23:46
issues in deep learning and neural networks. And I usually do the little challenge of changing a value of X original to highlight the fact that permute returns a different view of the original tensor. And a view in PyTorch shares memory with that original tensor. So if we change the value at zero, zero, zero of X original to, in my case, 728218, it happens the same value gets copied across to X permuted. So with that being said, we looked at selecting data from tensors here, and this is
3:24:20
using a technique called indexing. So let's just rehash that, because this is another thing that can be a little bit of a hurdle when first working with multi dimensional tensors. So let's see how we can select data from tensors with indexing. So if you've ever done indexing, indexing,
3:24:39
with PyTorch is similar to indexing with NumPy. If you've ever worked with NumPy, and you've done indexing, selecting data from arrays, NumPy uses an array as its main data type, PyTorch uses tensors. It's very similar. So let's again start by creating a tensor. And again, I'm just going to add a few code cells here, so I can make my screen right in the middle. Now we're going to import torch. Again, we don't need to import torch all the time,
3:25:10
just so you can run the notebook from here later on. X equals torch dot. Let's create a range again, just nice and simple. This is how I like to work out the fundamentals too, is just create the small range, reshape it, and the reshape has to be compatible with the original dimension. So we go one, three, three, and why is this because torch a range is going to return us nine values, because it's from the start here to the end minus one, and then one times three times three is what is
3:25:43
nine. So let's have a look x x dot shape. Beautiful. So we have one, two, three, four, five, six, seven, eight, nine of size one. So we have this is the outer bracket here, which is going to contain all of this. And then we have three, which is this one here, one, two, three. And then we have three, which is one, two, three. Now let's work with this. Let's index on our new tensor. So let's see what
3:26:19
happens when we get x zero, this is going to index on the first bracket. So we get this one here. So we've indexed on the first dimension here, the zero dimension on this one here, which is why we get what's inside here. And then let's try again, let's index on the middle bracket. So dimension one. So we got to go x, and then zero, and then zero. Let's see what happens there. Now is this the same as going x zero, zero? It is, there we go. So it depends on what you want to use. Sometimes
3:27:04
I prefer to go like this. So I know that I'm getting the first bracket, and then the zeroth version of that first bracket. So then we have these three values here. Now what do you think what's going to happen if we index on third dimension or the second dimension here? Well, let's find out. So let's index on the most in our bracket, which is last dimension.
3:27:31
So we have x zero, zero, zero. What numbers is going to give us back of x zero, on the zero dimension gives us back this middle tensor. And then if x zero, zero gives us back the zeroth index of the middle tensor. If we go x zero, zero, zero is going to give us the zeroth tensor, the zeroth index, and the zeroth element. A lot to take in there. But what we've done is we've just broken it down step by step. We've got this first zero targets this outer bracket
3:28:06
and returns us all of this. And then zero, zero targets this first because of this first zero, and then the zero here targets this. And then if we go zero, zero, zero, we target this, then we target this, and then we get this back because we are getting the zeroth index here. So if we change this to one, what do we get back? Two. And if we change these all to one, what will we get? This is a bit of trivia here, or a challenge. So we're going one, one, one.
3:28:45
Let's see what happens. Oh, no, did you catch that before I ran the code? I did that one quite quickly. We have index one is out of bounds. Why is that? Well, because this dimension is only one here. So we can only index on the zero. That's where it gets a little bit confusing because this says one, but because it's only got zero dimension, we can only index on the zero if to mention. But what if we do 011? What does that give us? Five. Beautiful. So I'd like to issue you the challenge
3:29:13
of how about getting number nine? How would you get number nine? So rearrange this code to get number nine. That's your challenge. Now, I just want to show you as well, is you can use, you can also use, you might see this, the semicolon to select all of a target dimension. So let's say we wanted to get all of the zeroth dimension, but the zero element from that. We can get 123. And then let's say we want to say get all values of the zeroth and first dimensions,
3:29:51
but only index one of the second dimension. Oh, that was a mouthful. But get all values of zeroth and first dimensions, but only index one of second dimension. So let's break this down step by step. We want all values of zeroth and first dimensions, but only index one of the second dimension. We press enter, shift enter, 258. So what did we get there? 258. Okay. So we've got all elements of the zeroth and first dimension, but then so which will return us this thing here.
3:30:30
But then we only want 258, which is the first element here of the second dimension, which is this three there. So quite confusing. But with some practice, you can figure out how to select almost any numbers you want from any kind of tensor that you have. So now let's try again, get all values of the zero dimension, but only the one index value of the first and second dimension. So what might this look like? Let's break it down again. So we come down here x,
3:31:05
and we're going to go all values of the zero dimension because zero comes first. And then we want only the one index value of the first and only the one index value of the second. What is this going to give us five? Oh, we selected the middle tensor. So really, this line of code is exactly the same as this line of code here, except we've got the square brackets on the outside here, because we've got this semicolon there. So if we change this to a zero,
3:31:34
we remove that. But because we've got the semicolon there, we've selected all the dimensions. So we get back the square bracket there, something to keep in mind. Finally, let's just go one more. So get index zero of zero and first dimension, and all values of second dimension. So x zero, zero. So zero, the index of zero and first dimension, zero, zero, and all values of the second dimension. What have we just done here? We've got tensor one,
3:32:11
two, three, lovely. This code again is equivalent to what we've done up here. This has a semicolon on the end. But what this line explicitly says without the semicolon is, hey, give us all the values on the remaining dimension there. So my challenge for you is to take this tensor that we have got here and index on it to return nine. So I'll write down here, index on x to return nine. So if you have a look at x, as well as index on x to return three, six, nine. So these values
3:32:54
here. So give those both a go and I'll see you in the next video. Welcome back. How'd you go? Did you give the challenge ago? I finished the last video with issuing the challenge to index on x to return nine and index on x to return three, six, nine. Now here's what I came up with. Again, there's a few different ways that you could approach both of these. But this is just what I've found. So because x is one, three, three of size, well, that's his dimensions. If we want to
3:33:25
select nine, we need zero, which is this first outer bracket to get all of these elements. And then we need two to select this bottom one here. And then we need this final two to select the second dimension of this bottom one here. And then for three, six, nine, we need all of the elements in the first dimension, all of the in the zeroth dimension, all of the elements in the first dimension. And then we get two, which is this three, six, nine set up here. So that's how I
3:33:56
would practice indexing, start with whatever shape tensor you like, create it something like this, and then see how you can write different indexing to select whatever number you pick. So now let's move on to the next part, which is PyTorch tensors and NumPy. So NumPy is a popular scientific, very popular. PyTorch actually requires NumPy when you install PyTorch. Popular scientific Python numerical computing library, that's a bit of a mouthful. And because of this,
3:34:37
PyTorch has functionality to interact with it. So quite often, you might start off with, let's change this into Markdown, you might start off with your data, because it's numerical format, you might start off with data in NumPy, NumPy array, want in PyTorch tensor. Because your data might be represented by NumPy because it started in NumPy, but say you want to do some deep learning on it and you want to leverage PyTorch's deep learning capabilities,
3:35:12
well, you might want to change your data from NumPy to a PyTorch tensor. And PyTorch has a method to do this, which is torch from NumPy, which will take in an ND array, which is NumPy's main data type, and change it into a torch tensor. We'll see this in a second. And then if you want to go from PyTorch tensor to NumPy because you want to use some sort of NumPy method, well, the method to do this is torch dot tensor, and you can call dot NumPy on it. But this is all
3:35:47
just talking about in words, let's see it in action. So NumPy array to tensor. Let's try this out first. So we'll import torch so we can run this cell on its own, and then import NumPy as np, the common naming convention for NumPy, we're going to create an array in NumPy. And we're going to just put one to eight, a range. And then we're going to go tensor equals torch from NumPy
3:36:20
because we want to go from NumPy array to a torch tensor. So we use from NumPy, and then we pass in array, and then we have array and tensor. Wonderful. So there's our NumPy array, and our torch tensor with the same data. But what you might notice here is that the D type for the tensor is torch dot float 64. Now why is this? It's because NumPy's default data type. Oh, D type is float 64. Whereas tensor, what have we discussed before? What's pytorch's default data type?
3:36:58
float 64. Well, that's not pytorch's default data type. If we were to create torch, a range, 1.0 to 8.0, by default, pytorch is going to create it in float 32. So just be aware of that. If you are going from NumPy to pytorch, the default NumPy data type is float 64. And pytorch reflects that data type when you use the from NumPy method. I wonder if there's a D type. Can we go D type equals torch dot float 32? Takes no keyword.
3:37:36
Okay. But how could we change the data type here? Well, we could go type torch float 32.
3:37:44
Yeah, that will give us a tensor D type of float 32 instead of float 64. Beautiful. I'll just keep that there so you know, warning when converting from NumPy pytorch, pytorch reflects NumPy's default data type of float 64, unless specified. Otherwise, because what have we discussed, when you're trying to perform certain calculations, you might run into a data type issue. So you might need to convert the type from float 64 to float 32. Now, let's see what happens. What do you think
3:38:32
will happen if we change the array? We change the value of an array. Well, let's find out. So change the value of array. The question is, what will this do to tensor? Because we've used the from NumPy method, do you think if we change the array, the tensor will change? So let's try this array equals array plus one. So we're just adding one to every value in the array. Now, what is the array and the tensor going to look like? Uh huh. So array, we only change the first
3:39:15
value there. Oh, sorry, we change every value because we have one to seven. Now it's two, three, four, five, six, seven, eight. We change the value from the array. It doesn't change the value of the tensor. So that's just something to keep in mind. If you use from NumPy, we get a new tensor in memory here. So the original, the new tensor doesn't change if you change the original array. So now let's go from tensor to NumPy. If you wanted to go back to NumPy,
3:39:43
tensor to NumPy array. So we'll start with a tensor. We could use the one we have right now, but we're going to create another one, but we'll create one of ones just for fun. One rhymes with fun. NumPy tensor equals. How do we go to NumPy? Well, we have torch dot tensor dot NumPy. So we just simply call NumPy on here. And then we have tensor and NumPy tensor. What data type do you think the NumPy tensor is going to have? Because we've returned it to NumPy. Pi torches, default data type is
3:40:21
Flight 32. So if we change that to NumPy, what's going to be the D type of the NumPy tensor? NumPy tensor dot D type. It reflects the original D type of what you set the tensor as. So just keep that in mind. If you're going between PyTorch and NumPy, default data type of NumPy is float 64, whereas the default data type of PyTorch is float 32. So that may cause some errors if you're doing different kinds of calculations. Now, what do you think is going to happen if we
3:40:51
went from our tensor to an array, if we change the tensor, change the tensor, what happens to
3:41:01
NumPy tensor? So we get tensor equals tensor plus one. And then we go NumPy tensor. Oh, we'll get tensor as well. So our tensor is now all twos because we added one to the ones. But our NumPy tensor remains the same. Remains unchanged. So this means they don't share memory. So that's how we go in between PyTorch and NumPy. If you'd like to look up more, I'd encourage you to go PyTorch and NumPy. So warm up NumPy, beginner. There's a fair few tutorials here on
3:41:40
PyTorch because NumPy is so prevalent, they work pretty well together. So have a look at that. There's a lot going on there. There's a few more links, I'd encourage you to check out, but we've covered some of the main ones that you'll see in practice. With that being said, let's now jump into the next video where we're going to have a look at the concept of reproducibility. If you'd like to look that up, I'd encourage you to search PyTorch's reproducibility and see
3:42:05
what you can find. Otherwise, I'll see you in the next video. Welcome back. It's now time for us to cover the topic of reproducibility. If I could even spell it, that would be fantastic. Reproducibility. Trying to take the random out of random. So we've touched upon the concept of neural networks harnessing the power of randomness. And what I mean by that is we haven't actually built our own neural network yet, but we will be doing that. And we've created tenses full of random
3:42:40
values. And so in short, how our neural network learns is start with random numbers, perform tensor operations, update random numbers to try and make them better representations of the data. Again, again, again, again, again. However, if you're trying to do reproducible experiments, sometimes you don't want so much randomness. And what I mean by this is if we were creating random tensors, from what we've seen so far is that every time we create a random tensor, let's create one here,
3:43:28
torch dot rand, and we'll create it of three three. Every time we run this cell, it gives us new numbers. So 7 7 5 2. There we go. Rand again. Right. So we get a whole bunch of random numbers here.
3:43:45
Every single time. But what if you were trying to share this notebook with a friend, so say you went up share and you clicked the share link and you sent that to someone and you're like, hey, try out this machine learning experiment I did. And you wanted a little less randomness because neural networks start with random numbers. How might you do that? Well, let's this write down to reduce the randomness in neural networks. And pytorch comes the concept of a
3:44:20
random seed. So we're going to see this in action. But essentially, let's write this down, essentially what the random seed does is flavor the randomness. So because of how computers work, they're actually not true randomness. And actually, there's arguments against this, and it's quite a big debate in the computer science topic, whatnot, but I am not a computer scientist, I am a machine learning engineer. So computers are fundamentally deterministic.
3:44:56
It means they run the same steps over and over again. So what the randomness we're doing here is referred to as pseudo randomness or generated randomness. And the random seed, which is what you see a lot in machine learning experiments, flavors that randomness. So let's see it in practice. And at the end of this video, I'll give you two resources that I'd recommend to learn a little bit more about the concept of pseudo randomness and reproducibility in pytorch.
3:45:22
Let's start by importing torch so you could start this notebook right from here. Create two random tensors. We'll just call this random tensor a equals torch dot rand and we'll go three four and we'll go random tensor b equals torch dot rand same size three four. And then if we have a look at let's go print random tensor a print random tensor b. And then let's print to see if they're equal anywhere random tensor a equals equals equals random tensor b. Now what do you
3:46:08
think this is going to do? If we have a look at one equals one, what does it return? True. So this is comparison operator to compare two different tensors. We're creating two random tensors here. We're going to have a look at them. We'd expect them to be full of random values. Do you think any of the values in each of these random tensors is going to be equal to each other?
3:46:31
Well, there is a chance that they are, but it's highly unlikely. I'll be quite surprised if they are. Oh, again, my connection might be a little bit. Oh, there we go. Beautiful. So we have tensor a tensor of three four with random numbers. And we have tensor b of three four with random numbers. So if we were, if I was to share this notebook with my friend or my colleague or even you, if you ran this cell, you are going to get random numbers as well. And you have every chance of
3:47:00
replicating one of these numbers. But again, it's highly unlikely. So again, I'm getting that automatic save failed. You might get that if your internet connection is dropping out, maybe that's something going on with my internet connection. But again, as we've seen, usually this resolves itself. If you try a few times, I'll just keep coding. If it really doesn't resolve itself, you can go file is a download notebook or save a copy and drive download. You can download the
3:47:26
notebook, save it to your local machine, re upload it to upload notebook and start again in another Google Colab instance. But there we go. It fixed itself. Wonderful troubleshooting on the fly. So the way we make these reproducible is through the concept of a random seed. So let's have a look at that. Let's make some random, but reproducible tenses. So import torch. And we're going to set the random seed by going torch dot manual seed random. Oh, we don't have random set yet.
3:48:14
I'm going to set my random seed. You set the random seed to some numerical value. 42 is a common one. You might see zero. You might see one, two, three, four. Essentially, you can set it to whatever you want. And each of these, you can think of 77, 100, as different flavors of randomness. So I like to use 42, because it's the answer to the universe. And then we go random seed. And now let's create some random tenses. Random tensor C with the flavor of our random seed. Three,
3:48:50
four. And then we're going to go torch tensor D equals torch dot rand three, four. Now, let's see what happens. We'll print out random tensor C. And we'll print out random tensor D. And then we'll print out to see if they're equal anywhere. Random tensor C equals random tensor D. So let's find out what happens. Huh, what gives? Well, we've got randomness. We set the random seed. We're telling pytorch a flavor our randomness with 42 torch manual seed. Hmm, let's try set the manual
3:49:42
seed each time we call a random method. We go there. Ah, much better. So now we've got some flavored randomness. So a thing to keep in mind is that if you want to use the torch manual seed, generally it only works for one block of code if you're using a notebook. So that's just something to keep in mind. If you're creating random tensors, one after the other, we're using assignment like this, you should use torch dot manual seed every time you want to call the rand
3:50:15
method or some sort of randomness. However, if we're using other torch processes, usually what you might see is torch manual seed is set right at the start of a cell. And then a whole bunch of code is done down here. But because we're calling subsequent methods here, we have to reset the random seed. Otherwise, if we don't do this, we comment this line, it's going to flavor the randomness of torch random tensor C with torch manual seed. But then random tensor D is just
3:50:42
going to have no flavor. It's not going to use a random seed. So we reset it there. Wonderful. So I wonder, does this have a seed method? Let's go torch dot rand. Does this have seed? Sometimes they have a seed method. Seed, no, it doesn't. Okay, that's all right. The more you learn, but there's documentation for torch dot rand. And I said that I was going to link at the end of this video. So the manual seed is a way to, or the random seed, but in
3:51:14
torch, it's called a manual seed is a way to flavor the randomness. So these numbers, as you see, are still quite random. But the random seed just makes them reproducible. So if I was to share this with you, if you had to run this block of code, ideally, you're going to get the same numerical output here. So with that being said, I'd like to refer to you to the pie torch reproducibility document, because we've only quite scratched the surface of this of reproducibility. We've covered
3:51:40
one of the main ones. But this is a great document on how to go through reproducibility in pie torch. So this is your extra curriculum for this, even if you don't understand what's going on in a lot of the code here, just be aware of reproducibility, because it's an important topic in machine learning and deep learning. So I'll put this here, extra resources for reproducibility. As we go pie torch randomness, we'll change this into markdown. And then finally, the concept
3:52:14
of a random seed is Wikipedia random seed. So random seeds quite a universal concept, not just for pie torch, there's a random seed and NumPy as well. So if you'd like to see what this means, yeah, initialize a pseudo random number generator. So that's a big word, pseudo random number generator. But if you'd like to learn about more random number generation computing, and what a random seed does is I'd refer to you to check out this documentation here.
3:52:44
Whoo, far out, we have covered a lot. But there's a couple more topics you should really be aware of to finish off the pie torch fundamentals. You got this. I'll see you in the next video. Welcome back. Now, let's talk about the important concept of running tenses or pie torch objects. So running tenses and pie torch objects on GPUs and making faster computations. So we've discussed that GPUs, let me just scroll down a little bit here, GPUs equal faster
3:53:26
computation on numbers. Thanks to CUDA plus NVIDIA hardware plus pie torch working behind the scenes to make everything hunky dory. Good. That's what hunky dory means, by the way, if you never heard that before. So let's have a look at how we do this. Now, we first need to talk about, let's go here one getting a GPU. There's a few different ways we've seen one before. Number one easiest is to use what we're using right now. Use Google Colab for a free GPU.
3:54:13
But there's also Google Colab Pro. And I think there might even be, let's look up Google Colab Pro. Choose the best that's right for you. I use Google Colab Pro because I use it almost every day. So yeah, I pay for Colab Pro. You can use Colab for free, which is might be what you're using. There's also Colab Pro Plus, which has a lot more advantages as well. But Colab Pro is giving me faster GPUs, so access to faster GPUs, which means you spend less time waiting while your code is running.
3:54:45
More memory, longer run time, so it'll last a bit longer if you leave it running idle. And then Colab Pro again is a step up from that. I personally haven't had a need yet to use Google Colab Pro Plus. You can complete this whole course on the free tier as well. But as you start to code more, as you start to run bigger models, as you start to want to compute more, you might want to look into something like Google Colab Pro. Or let's go here. Options to upgrade as well.
3:55:14
And then another way is use your own GPU. Now this takes a little bit of setup and requires the investment of purchasing a GPU. There's lots of options. So one of my favorite posts for getting a GPU is, yeah, the best GPUs for deep learning in 2020, or something like this. What do we got? Deep learning? Tim Detmos. This is, yeah, which GPUs to get for deep learning? Now, I believe at the time of this video, I think it's been updated since this date. So don't take
3:56:07
my word for it. But this is a fantastic blog post for figuring out what GPUs see this post for what option to get. And then number three is use cloud computing. So such as GCP, which is Google Cloud Platform AWS, which is Amazon Web Services or Azure. These services, which is Azure is by Microsoft, allow you to rent computers on the cloud and access them. So the first option using Google Colab, which is what we're using is by far the easiest
3:56:51
and free. So there's big advantages there. However, the downside is that you have to use a website here, Google Colab, you can't run it locally. You don't get the benefit of using cloud computing, but my personal workflow is I run basically all of my small scale experiments and things like learning new stuff in Google Colab. And then if I want to upgrade things, run video experiments, I have my own dedicated deep learning PC, which I have built with a big powerful GPU. And then
3:57:18
also I use cloud computing if necessary. So that's my workflow. Start with Google Colab. And then these two, if I need to do some larger experiments. But because this is the beginning of course, we can just stick with Google Colab for the time being. But I thought I'd make you aware of these other two options. And if you'd like to set up a GPU, so four, two, three, PyTorch plus GPU drivers, which is CUDA takes a little bit of setting up to do this, refer to PyTorch
3:57:56
setup documentation. So if we go to pytorch.org, they have some great setup guides here, get started. And we have start locally. This is if you want to run on your local machine, such as a Linux setup. This is what I have Linux CUDA 11.3. It's going to give you a conda install command to use conda. And then if you want to use cloud partners, which is Alibaba Cloud, Amazon Web Services, Google Cloud Platform, this is where you'll want to go. So I'll just link
3:58:31
this in here. But for this course, we're going to be focusing on using Google Colab. So now, let's see how we might get a GPU in Google Colab. And we've already covered this, but I'm going to recover it just so you know. We're going to change the runtime type. You can go in any notebook and do this, runtime type, hardware accelerator, we can select GPU, click save. Now this is going to restart our runtime and connect us to our runtime, aka a Google compute instance with a GPU. And so
3:59:07
now if we run NVIDIA SMI, I have a Tesla P100 GPU. So let's look at this Tesla P100
3:59:21
GPU. Do we have an image? Yeah, so this is the GPU that I've got running, not the Tesla car, the GPU. So this is quite a powerful GPU. That is because I have upgraded to Colab Pro. Now, if you're not using Colab Pro, you might get something like a Tesla K80, which is a slightly less powerful GPU than a Tesla P100, but still a GPU nonetheless and will still work faster than just running PyTorch code on the pure CPU, which is the default in Google Colab and the default
3:59:53
in PyTorch. And so now we can also check to see if we have GPU access with PyTorch. So let's go here. This is number two now. Check for GPU access with PyTorch. So this is a little command that's going to allow us or tell us if PyTorch, just having the GPU here, this is by the way, another thing that Colab has a good setup with, is that all the connections between PyTorch and the NVIDIA GPU are set up for us. Whereas when you set it up on your own GPU or using cloud computing,
4:00:34
there are a few steps you have to go through, which we're not going to cover in this course. I'd highly recommend you go through the getting started locally set up if you want to do that, to connect PyTorch to your own GPU. So let's check for the GPU access with PyTorch. This is another advantage of using Google Colab. Almost zero set up to get started. So import torch and then we're going to go torch dot cuda dot is available. And remember, cuda is
4:01:08
NVIDIA's programming interface that allows us to use GPUs for numerical computing. There we go, beautiful. So big advantage of Google Colab is we get access to a free GPU. In my case, I'm paying for the faster GPU, but in your case, you're more than welcome to use the free version. All that means it'll be slightly slower than a faster GPU here. And we now have access to GPUs with PyTorch. So there is one more thing known as device agnostic code. So set up device agnostic
4:01:44
code. Now, this is an important concept in PyTorch because wherever you run PyTorch, you might not always have access to a GPU. But if there was access to a GPU, you'd like it to use it if it's available. So one of the ways that this is done in PyTorch is to set the device variable. Now, really, you could set this to any variable you want, but you're going to see it used as device quite often. So cuda if torch dot cuda is available. Else CPU. So all this is going to say, and we'll
4:02:21
see where we use the device variable later on is set the device to use cuda if it's available. So it is so true. If it's not available, if we don't have access to a GPU that PyTorch can use, just default to the CPU. So with that being said, there's one more thing. You can also count the number of GPUs. So this won't really apply to us for now because we're just going to stick with using one GPU. But as you upgrade your PyTorch experiments and machine learning experiments,
4:02:51
you might have access to more than one GPU. So you can also count the devices here.
4:02:57
We have access to one GPU, which is this here. So the reason why you might want to count the number of devices is because if you're running huge models on large data sets, you might want to run one model on a certain GPU, another model on another GPU, and so on and so on. But final thing before we finish this video is if we go PyTorch device agnostic code, cuda semantics, there's a little section in here called best practices. This is basically what we just covered there is setting
4:03:31
the device argument. Now this is using the arg pass, but so yeah, there we go. args.device, torch.device, cuda, args.device, torch.device, CPU. So this is one way to set it from the Python arguments when you're running scripts, but we're using the version of running it through a notebook.
4:03:51
So check this out. I'll just link this here, device agnostic code. It's okay if you're not sure of what's going on here. We're going to cover it a little bit more later on throughout the course, but right here for PyTorch, since it's capable of running compute on the GPU or CPU, it's best practice to set up device agnostic code, e.g. run on GPU if available,
4:04:29
else default to CPU. So check out the best practices for using cuda, which is namely setting up device agnostic code. And let's in the next video, see what I mean about setting our PyTorch tensors and objects to the target device. Welcome back. In the last video, we checked out a few different options for getting a GPU, and then getting PyTorch to run on the GPU. And for now we're using Google Colab, which is the easiest way to get set up because it gives us free access to a GPU,
4:05:04
faster ones if you set up with Colab Pro, and it comes with PyTorch automatically set up to use the GPU if it's available. So now let's see how we can actually use the GPU. So to do so, we'll look at putting tensors and models on the GPU. So the reason we want our tensors slash models on the GPU is because using GPU results in faster computations. And if we're getting our machine learning models to find patterns and numbers, GPUs are great at doing numerical calculations.
4:05:48
And the numerical calculations we're going to be doing are tensor operations like we saw above. So the tensor operations, well, we've covered a lot. Somewhere here, tensor operations, there we go, manipulating tensor operations. So if we can run these computations faster, we can discover patterns in our data faster, we can do more experiments, and we can work towards finding the best possible model for whatever problem that we're working on. So let's see,
4:06:15
we'll create a tensor, as usual, create a tensor. Now the default is on the CPU. So tensor equals torch dot tensor. And we'll just make it a nice simple one, one, two, three. And let's write here, tensor not on GPU will print out tensor. And this is where we can use, we saw this parameter before device. Can we pass it in here? Device equals CPU. Let's see what this comes out with. There we go. So if we print it out, tensor 123 is on the CPU.
4:06:54
But even if we got rid of that device parameter, by default, it's going to be on the CPU. Wonderful. So now PyTorch makes it quite easy to move things to, and I'm saying to for a reason, to the GPU, or to, even better, the target device. So if the GPU is available, we use CUDA. If it's not, it uses CPU. This is why we set up the device variable. So let's see, move tensor to GPU. If available, tensor on GPU equals tensor dot two device. Now let's have a look at this, tensor on GPU.
4:07:43
So this is going to shift the tensor that we created up here to the target device.
4:07:50
Wonderful. Look at that. So now our tensor 123 is on device CUDA zero. Now this is the index of the GPU that we're using, because we only have one, it's going to be at index zero. So later on, when you start to do bigger experiments and work with multiple GPUs, you might have different tensors that are stored on different GPUs. But for now, we're just sticking with one GPU, keeping it nice and simple. And so you might have a case where you want to move, oh, actually, the reason why we
4:08:18
set up device agnostic code is again, this code would work if we run this, regardless if we had, so it won't error out. But regardless if we had a GPU or not, this code will work. So whatever device we have access to, whether it's only a CPU or whether it's a GPU, this tensor will move to whatever target device. But since we have a GPU available, it goes there. You'll see this a lot. This two method moves tensors and it can be also used for models. We're going to see that later on. So just
4:08:50
keep two device in mind. And then you might want to, for some computations, such as using NumPy, NumPy only works with the CPU. So you might want to move tensors back to the CPU, moving tensors back to the CPU. So can you guess how we might do that? It's okay if you don't know. We haven't covered a lot of things, but I'm going to challenge you anyway, because that's the fun part of thinking about something. So let's see how we can do it. Let's write down if tensor is on GPU, can't transform
4:09:27
it to NumPy. So let's see what happens if we take our tensor on the GPU and try to go NumPy. What happens? Well, we get an error. So this is another huge error. Remember the top three errors in deep learning or pytorch? There's lots of them, but number one, shape errors, number two, data type issues. And with pytorch, number three is device issues. So can't convert CUDA zero device type tensor to NumPy. So NumPy doesn't work with the GPU. Use tensor dot CPU
4:09:58
to copy the tensor to host memory first. So if we call tensor dot CPU, it's going to bring our target tensor back to the CPU. And then we should be able to use it with NumPy. So to fix the GPU tensor with NumPy issue, we can first set it to the CPU. So tensor back on CPU
4:10:27
equals tensor on GPU dot CPU. We're just taking what this said here. That's a beautiful thing about pytorch is very helpful error messages. And then we're going to go NumPy. And then if we go tensor back on CPU, is this going to work? Let's have a look. Oh, of course, it's not because I typed it wrong. And I've typed it again twice. Third time, third time's a charm.
4:10:54
There we go. Okay, so that works because we've put it back to the CPU first before calling NumPy. And then if we refer back to our tensor on the GPU, because we've reassociated this, again, we've got typos galore classic, because we've reassigned tensor back on CPU, our tensor on GPU remains unchanged. So that's the four main things about working with pytorch on the GPU. There are a few more tidbits such as multiple GPUs, but now you've got the fundamentals. We're
4:11:26
going to stick with using one GPU. And if you'd like to later on once you've learned a bit more research into multiple GPUs, well, as you might have guessed, pytorch has functionality for that too. So have a go at getting access to a GPU using colab, check to see if it's available, set up device agnostic code, create a few dummy tensors and just set them to different devices, see what happens if you change the device parameter, run a few errors by trying to do some NumPy calculations
4:11:53
with tensors on the GPU, and then bring those tensors on the GPU back to NumPy and see what happens there. So I think we've covered, I think we've reached the end of the fundamentals. We've covered a fair bit. Introduction to tensors, the minmax, a whole bunch of stuff inside the introduction to tensors, finding the positional minmax, reshaping, indexing, working with tensors and NumPy, reproducibility, using a GPU and moving stuff back to the GPU far out. Now you're probably wondering,
4:12:24
Daniel, we've covered a whole bunch. What should I do to practice all this? Well, I'm glad you asked. Let's cover that in the next video. Welcome back. And you should be very proud of your self right now. We've been through a lot, but we've covered a whole bunch of PyTorch fundamentals. These are going to be the building blocks that we use throughout the rest of the course. But before moving on to the next section, I'd encourage you to try out what you've learned
4:12:51
through the exercises and extra curriculum. Now, I've set up a few exercises here based off everything that we've covered. If you go into learn pytorch.io, go to the section that we're currently on. This is going to be the case for every section, by the way. So just keep this in mind, is we're working on PyTorch fundamentals. Now, if you go to the PyTorch fundamentals notebook, this is going to refresh, but that if you scroll down to the table of contents at the bottom of
4:13:20
each one is going to be some exercises and extra curriculum. So these exercises here, such as documentation reading, because a lot you've seen me refer to the PyTorch documentation for almost everything we've covered a lot, but it's important to become familiar with that. So exercise number one is read some of the documentation. Exercise number two is create a random tensor with shape, seven, seven. Three, perform a matrix multiplication on the tensor from two
4:13:48
with another random tensor. So these exercises are all based off what we've covered here. So I'd encourage you to reference what we've covered in whichever notebook you choose, could be this learn pytorch.io, could be going back through the one we've just coded together in the video. So I'm going to link this here, exercises, see exercises for this notebook here.
4:14:16
So then how should you approach these exercises? So one way would be to just read them here, and then in collab we'll go file new notebook, wait for the notebook to load. Then you could call this zero zero pytorch exercises or something like that, and then you could start off by importing torch, and then away you go. For me, I'd probably set this up on one side of the screen, this one up on the other side of the screen, and then I just have the exercises here. So number one,
4:14:51
I'm not going to really write much code for that, but you could have documentation reading here.
4:14:57
And then so this encourages you to read through torch.tensor and go through there
4:15:04
for 10 minutes or so. And then for the other ones, we've got create a random tensor with shape seven seven. So we just comment that out. So torch, round seven seven, and there we go. Some are as easy as that. Some are a little bit more complex. As we go throughout the course, these exercises are going to get a little bit more in depth as we've learned more.
4:15:26
But if you'd like an exercise template, you can come back to the GitHub. This is the home for all of the course materials. You can go into extras and then exercises. I've created templates for each of the exercises. So pytorch fundamentals exercises. If you open this up, this is a template for all of the exercises. So you see there, create a random tensor with shape seven seven. These are all just headings. And if you'd like to open this in CoLab and work on it,
4:15:55
how can you do that? Well, you can copy this link here. Come to Google CoLab. We'll go file, open notebook, GitHub. You can type in the link there. Click search. What's this going to do? Boom. Pytorch fundamentals exercises. So now you can go through all of the exercises. This will be the same for every module on the course and test your knowledge. Now it is open book. You can use the notebook here, the ones that we've coded together. But I would encourage you to try
4:16:30
to do these things on your own first. If you get stuck, you can always reference back. And then if you'd like to see an example solutions, you can go back to the extras. There's a solutions folder as well. And that's where the solutions live. So the fundamental exercise solutions. But again, I would encourage you to try these out, at least give them a go before having a look at the solutions. So just keep that in mind at the end of every module, there's exercises and extra curriculum.
4:16:58
The exercises will be code based. The extra curriculum is usually like reading based. So spend one hour going through the Pytorch basics tutorial. I recommend the quick start in tensor sections. And then finally to learn more on how a tensor can represent data, watch the video what's a tensor which we referred to throughout this. But massive effort on finishing the Pytorch fundamentals section. I'll see you in the next section. Friends, welcome back to
4:17:31
the Pytorch workflow module. Now let's have a look at what we're going to get into. So this is a Pytorch workflow. And I say a because it's one of many. When you get into deep learning machine learning, you'll find that there's a fair few ways to do things. But here's the rough outline of what we're going to do. We're going to get our data ready and turn it into tensors because remember a tensor can represent almost any kind of data. We're going to pick or
4:17:56
build or pick a pre-trained model. We'll pick a loss function and optimize it. Don't worry if you don't know what they are. We're going to cover this. We're going to build a training loop, fit the model to make a prediction. So fit the model to the data that we have. We'll learn how to evaluate our models. We'll see how we can improve through experimentation and we'll save and reload our trained model. So if you wanted to export your model from a notebook and use it
4:18:19
somewhere else, this is what you want to be doing. And so where can you get help? Probably the most important thing is to follow along with the code. We'll be coding all of this together. Remember model number one. If and out, run the code. Try it for yourself. That's how I learn best. Is I write code? I try it. I get it wrong. I try again and keep going until I get it right. Read the doc string because that's going to show you some documentation about the functions that
4:18:46
we're using. So on a Mac, you can use shift command and space in Google Colab or if you're on a Windows PC, it might be control here. If you're still stuck, try searching for it. You'll probably come across resources such as stack overflow or the PyTorch documentation. We've already seen this a whole bunch and we're probably going to see it a lot more throughout this entire course actually because that's going to be the ground truth of everything PyTorch. Try again. And finally,
4:19:11
if you're still stuck, ask a question. So the best place to ask a question will be at the PyTorch deep learning slash discussions tab. And then if we go to GitHub, that's just under here. So Mr. Deeburg PyTorch deep learning. This is all the course materials. We see here, this is your ground truth for the entire course. And then if you have a question, go to the discussions tab, new discussion, you can ask a question there. And don't forget to
4:19:36
please put the video and the code that you're trying to run. That way we can reference what's going on and help you out there. And also, don't forget, there is the book version of the course. So learn pytorch.io. By the time you watch this video, it'll probably have all the chapters here. But here's what we're working through. This is what the videos are based on. All of this, we're going to go through all of this. How fun is that? But this is just reference material.
4:20:00
So you can read this at your own time. We're going to focus on coding together. And speaking of coding.
4:20:09
Let's code. I'll see you over at Google Colab.
4:20:14
Oh, right. Well, let's get hands on with some code. I'm going to come over to colab.research.google.com. You may already have that bookmark. And I'm going to start a new notebook. So we're going to do everything from scratch here. We'll let this load up. I'm just going to zoom in a little bit.
4:20:35
Beautiful. And now I'm going to title this 01 pytorch workflow. And I'm going to put the video ending on here so that you know that this notebook's from the video. Why is that? Because in the course resources, we have the original notebook here, which is what this video notebook is going to be based off. You can refer to this notebook as reference for what we're going to go through. It's got a lot of pictures and beautiful text annotations. We're going to be focused on the
4:21:03
code in the videos. And then of course, you've got the book version of the notebook as well, which is just a different formatted version of this exact same notebook. So I'm going to link both of these up here. So let's write in here, pytorch workflow. And let's explore an example, pytorch end to end workflow. And then I'm going to put the resources. So ground truth notebook. We go here. And I'm also going to put the book version.
4:21:44
Book version of notebook. And finally, ask a question, which will be where at the discussions page. Then we'll go there. Beautiful. Let's turn this into markdown. So let's get started. Let's just jump right in and start what we're covering. So this is the trend I want to start getting towards is rather than spending a whole bunch of time going through keynotes and slides, I'd rather we just code together. And then we explain different things as they need to be
4:22:19
explained because that's what you're going to be doing if you end up writing a lot of pytorch is you're going to be writing code and then looking things up as you go. So I'll get out of these extra tabs. I don't think we need them. Just these two will be the most important. So what we're covering, let's create a little dictionary so we can check this if we wanted to later on. So referring to our pytorch workflows, at least the example one that we're going to go through,
4:22:45
which is just here. So we're going to go through all six of these steps, maybe a little bit of each one, but just to see it going from this to this, that's what we're really focused on. And then we're going to go through through rest the course like really dig deep into all of these. So what we're covering number one is data preparing and loading. Number two is we're going to see how we can build a machine learning model in pytorch or a deep learning model. And then we're going
4:23:16
to see how we're going to fit our model to the data. So this is called training. So fit is another word. As I said in machine learning, there's a lot of different names for similar things, kind of confusing, but you'll pick it up with time. So we're going to once we've trained a model, we're going to see how we can make predictions and evaluate those predictions, evaluating a model. If you make predictions, it's often referred to as inference. I typically
4:23:43
say making predictions, but inference is another very common term. And then we're going to look at how we can save and load a model. And then we're going to put it all together. So a little bit different from the visual version we have of the pytorch workflow. So if we go back to here, I might zoom in a little. There we go. So we're going to focus on this one later on, improve through experimentation. We're just going to focus on the getting data ready,
4:24:12
building a model, fitting the model, evaluating model, save and reload. So we'll see this one more, like in depth later on, but I'll hint at different things that you can do for this while we're working through this workflow. And so let's put that in here. And then if we wanted to refer to this later, we can just go what we're covering.
4:24:34
Oh, this is going to connect, of course. Beautiful. So we can refer to this later on, if we wanted to. And we're going to start by import torch. We're going to get pytorch ready to go import nn. So I'll write a note here. And then we haven't seen this one before, but we're going to see a few things that we haven't seen, but that's okay. We'll explain it as we go. So nn contains all of pytorch's building blocks for neural networks.
4:25:03
And how would we learn more about torch nn? Well, if we just go torch.nn, here's how I'd learn about it, pytorch documentation. Beautiful. Look at all these. These are the basic building blocks for graphs. Now, when you see the word graph, it's referring to a computational graph, which is in the case of neural networks, let's look up a photo of a neural network. Images, this is a graph. So if you start from here, you're going to go towards the right.
4:25:33
There's going to be many different pictures. So yeah, this is a good one. Input layer. You have a hidden layer, hidden layer to output layer. So torch and n comprises of a whole bunch of different layers. So you can see layers, layers, layers. And each one of these, you can see input layer, hidden layer one, hidden layer two. So it's our job as data scientists and machine learning engineers to combine these torch dot nn building blocks to build things such as these.
4:26:03
Now, it might not be exactly like this, but that's the beauty of pytorch is that you can combine these in almost any different way to build any kind of neural network you can imagine.
4:26:14
And so let's keep going. That's torch nn. We're going to get hands on with it, rather than just talk about it. And we're going to need map plot lib because what's our other motto? Our data explorers motto is visualize, visualize, visualize. And let's check our pytorch version. Pytorch version torch dot version. So this is just to show you you'll need
4:26:39
at least this version. So 1.10 plus CUDA 111. That means that we've got CU stands for CUDA. That means we've got access to CUDA. We don't have a GPU on this runtime yet, because we haven't gone to GPU. We might do that later.
4:26:56
So if you have a version that's lower than this, say 1.8, 0.0, you'll want pytorch 1.10 at least. If you have a version higher than this, your code should still work. But that's about enough for this video. We've got our workflow ready to set up our notebook, our video notebook. We've got the resources. We've got what we're covering. We've got our dependencies. Let's in the next one get started on one data, preparing and loading.
4:27:26
I'll see you in the next video.
4:27:29
Let's now get on to the first step of our pytorch workflow. And that is data, preparing and loading. Now, I want to stress data can be almost anything in machine learning. I mean, you could have an Excel spreadsheet, which is rows and columns, nice and formatted data. You could have images of any kind. You could have videos. I mean, YouTube has lots of data. You could have audio like songs or podcasts. You could have even DNA
4:28:09
these days. Patents and DNA are starting to get discovered by machine learning. And then, of course, you could have text like what we're writing here. And so what we're going to be focusing on throughout this entire course is the fact that machine learning is a game of two parts. So one, get data into a numerical representation to build a model to learn patterns in that numerical representation. Of course, there's more around it. Yes, yes, yes. I understand you can
4:28:47
get as complex as you like, but these are the main two concepts. And machine learning, when I say machine learning, saying goes for deep learning, you need some kind of, oh, number a call. Number a call. I like that word, number a call representation. Then you want to build a model to learn patterns in that numerical representation. And if you want, I've got a nice pretty picture that describes that machine learning a game of two parts. Let's refer to our data. Remember, data can be almost
4:29:16
anything. These are our inputs. So the first step that we want to do is create some form of numerical encoding in the form of tenses to represent these inputs, how this looks will be dependent on the data, depending on the numerical encoding you choose to use. Then we're going to build some sort of neural network to learn a representation, which is also referred to as patterns features or weights within that numerical encoding. It's going to output that
4:29:45
representation. And then we want to do something without representation, such as in the case of this, we're doing image recognition, image classification, is it a photo of Raman or spaghetti? Is this tweet spam or not spam? Is this audio file saying what it says here? I'm not going to say this because my audio assistant that's also named to this word here is close by and I don't want it to go off. So this is our game of two parts. One here is convert our data into a numerical
4:30:16
representation. And two here is build a model or use a pre trained model to find patterns in that numerical representation. And so we've got a little stationary picture here, turn data into numbers, part two, build a model to learn patterns in numbers. So with that being said, now let's create some data to showcase this. So to showcase this, let's create some known
4:30:49
data using the linear regression formula. Now, if you're not sure what linear regression is, or the formula is, let's have a look linear regression formula. This is how I'd find it. Okay, we have some fancy Greek letters here. But essentially, we have y equals a function of x and b plus epsilon. Okay. Well, there we go. A linear regression line has the equation in the form of y equals a plus bx. Oh, I like this one better. This is nice and simple. We're going to
4:31:20
start from as simple as possible and work up from there. So y equals a plus bx, where x is the explanatory variable, and y is the dependent variable. The slope of the line is b. And the slope is also known as the gradient. And a is the intercept. Okay, the value of when y when x equals zero. Now, this is just text on a page. This is formula on a page. You know how I like to learn things? Let's code it out. So let's write it here. We'll use a linear regression formula
4:31:59
to make a straight line with known parameters. I'm going to write this down because parameter
4:32:10
is a common word that you're going to hear in machine learning as well. So a parameter is something that a model learns. So for our data set, if machine learning is a game of two parts, we're going to start with this. Number one is going to be done for us, because we're going to start with a known representation, a known data set. And then we want our model to learn that representation. This is all just talk, Daniel, let's get into coding. Yes, you're right. You're
4:32:40
right. Let's do it. So create known parameters. So I'm going to use a little bit different names to what that Google definition did. So weight is going to be 0.7 and bias is going to be 0.3. Now weight and bias are another common two terms that you're going to hear in neural networks. So just keep that in mind. But for us, this is going to be the equivalent of our weight will be B and our bias will be A. But forget about this for the time being. Let's just focus on the code.
4:33:15
So we know these numbers. But we want to build a model that is able to estimate these numbers.
4:33:23
How? By looking at different examples. So let's create some data here. We're going to create a range of numbers. Start equals zero and equals one. We're going to create some numbers between zero and one. And they're going to have a gap. So the step the gap is going to be 0.02. Now we're going to create an X variable. Why is X a capital here?
4:33:47
Well, it's because typically X in machine learning you'll find is a matrix or a tensor. And if we remember back to the fundamentals, a capital represents a matrix or a tensor and a lowercase represents a vector. But now case it's going to be a little confusing because X is a vector. But later on, X will start to be a tensor and a matrix. So for now, we'll just keep the capital, not capital notation.
4:34:15
We're going to create the formula here, which is remember how I said our weight is in this case, the B and our bias is the A. So we've got the same formula here. Y equals weight times X plus bias. Now let's have a look at these different numbers. So we'll view the first 10 of X and we'll view the first 10 of Y. We'll have a look at the length of X and we'll have a look at the length of Y. Wonderful. So we've got some values here. We've got 50 numbers of each. This is a little
4:34:55
confusing. Let's just view the first 10 of X and Y first. And then we can have a look at the length here. So what we're going to be doing is building a model to learn some values,
4:35:12
to look at the X values here and learn what the associated Y value is and the relationship between those. Of course, we know what the relationship is between X and Y because we've coded this formula here. But you won't always know that in the wild. That is the whole premise of machine learning. This is our ideal output and this is our input. The whole premise of machine learning is to learn a representation of the input and how it maps to the output. So here are our
4:35:44
input numbers and these are our output numbers. And we know that the parameters of the weight and bias are 0.7 and 0.3. We could have set these to whatever we want, by the way. I just like the number 7 and 3. You could set these to 0.9, whatever, whatever. The premise would be the same. So, oh, and what I've just done here, I kind of just coded this without talking. But I just did torch a range and it starts at 0 and it ends at 1 and the step is 0.02. So there
4:36:14
we go, 000 by 0.02, 04. And I've unsqueezed it. So what does unsqueezed do? Removes the extra dimensions. Oh, sorry, ads are extra dimension. You're getting confused here. So if we remove that,
4:36:31
we get no extra square bracket. But if we add unsqueeze, you'll see that we need this later on for when we're doing models. Wonderful. So let's just leave it at that. That's enough for this video, we've got some data to work with. Don't worry if this is a little bit confusing for now, we're going to keep coding on and see what we can do to build a model to infer patterns in this data. But right now, I want you to have a think, this is tensor data, but it's just numbers on a
4:36:58
page. What might be a better way to hint, this is a hint by the way, visualize it. What's our data explorer's motto? Let's have a look at that in the next video. Welcome back. In the last video, we created some numbers on a page using the linear regression formula with some known parameters. Now, there's a lot going on here, but that's all right. We're going to keep building upon what we've done and learn by doing. So in this video, we're going to cover one of the most
4:37:28
important concepts in machine learning in general. So splitting data into training and test sets. One of the most important concepts in machine learning in general. Now, I know I've said this already a few times. One of the most important concepts, but truly, this is possibly, in terms of data, this is probably the number one thing that you need to be aware of. And if you've come from a little bit of a machine learning background, you probably well and truly know all about this.
4:38:02
But we're going to recover it anyway. So let's jump in to some pretty pictures. Oh, look at that one speaking of pretty pictures. But that's not what we're focused on now. We're looking at the three data sets. And I've written down here possibly the most important concept in machine learning, because it definitely is from a data perspective. So the course materials, imagine you're at university. So this is going to be the training set. And then you have the
4:38:29
practice exam, which is the validation set. Then you have the final exam, which is the test set. And the goal of all of this is for generalization. So let's step back. So say you're trying to learn something at university or through this course, you might have all of the materials, which is your training set. So this is where our model learns patterns from. And then to practice what you've done, you might have a practice exam. So the mid semester exam or something like that. Now,
4:39:01
let's just see if you're learning the course materials well. So in the case of our model, we might tune our model on this plastic exam. So we might find that on the validation set, our model doesn't do too well. And we adjusted a bit, and then we retrain it, and then it does better. Before finally, at the end of semester, the most important exam is your final exam. And this is to see if you've gone through the entire course materials, and you've learned some things.
4:39:32
Now you can adapt to unseen material. And that's a big point here. We're going to see this in practice is that when the model learns something on the course materials, it never sees the validation set or the test set. So say we started with 100 data points, you might use 70 of those data points for the training material. You might use 15% of those data points, so 15 for the practice. And you might use 15 for the final exam. So this final exam is just like if you're at university
4:40:03
learning something is to see if, hey, have you learned any skills from this material at all? Are you ready to go into the wild into the quote unquote real world? And so this final exam is to test your model's generalization, because it's never seen this data is, let's define generalization is the ability for a machine learning model or a deep learning model to perform well on data it hasn't seen before, because that's our whole goal, right? We want to build a machine learning model
4:40:32
on some training data that we can deploy in our application or production setting. And then more data comes in that it hasn't seen before. And it can make decisions based on that new data because of the patterns it's learned in the training set. So just keep this in mind, three data sets training validation test. And if we jump in to the learn pytorch book, we've got split data. So we're going to create three sets. Or in our case, we're only going to
4:41:04
create two or training in a test. Why is that? Because you don't always need a validation set. There is often a use case for a validation set. But the main two that are always used is the training set and the testing set. And how much should you split? Well, usually for the training set, you'll have 60 to 80% of your data. If you do create a validation set, you'll have somewhere between 10 and 20. And if you do create a testing set, it's a similar split to the validation set,
4:41:33
you'll have between 10 and 20%. So training, always testing always validation often, but not always. So with that being said, I'll let you refer to those materials if you want. But now let's create a training and test set with our data. So we saw before that we have 50 points, we have X and Y, we have one to one ratio. So one value of X relates to one value of Y. And we know that the split now for the training set is 60 to 80%. And the test set is 10 to 20%.
4:42:09
So let's go with the upper bounds of each of these, 80% and 20%, which is a very common split, actually 80, 20. So let's go create a train test split. And we're going to go train split. We'll create a number here so we can see how much. So we want an integer of 0.8, which is 80% of the length of X. What does that give us? Train split should be about 40 samples. Wonderful. So we're going to create 40 samples of X and 40 samples of Y. Our model will train on those 40
4:42:46
samples to predict what? The other 10 samples. So let's see this in practice. So X train,
4:42:55
Y train equals X. And we're going to use indexing to get all of the samples up until the train split. That's what this colon does here. So hey, X up until the train split, Y up until the train split, and then for the testing. Oh, thanks for that. Auto correct cola, but didn't actually need that one. X test. Y test equals X. And then we're going to get everything from the train split onwards. So the index onwards, that's what this notation means here. And Y from the train split onwards as
4:43:36
well. Now, there are many different ways to create a train and test split. Ours is quite simple here, but that's because we're working with quite a simple data set. One of the most popular methods that I like is scikit learns train test split. We're going to see this one later on. It adds a little bit of randomness into splitting your data. But that's for another video, just to make you aware of it. So let's go length X train. We should have 40 training samples to
4:44:09
how many testing samples length X test and length Y test. Wonderful 40 40 10 10 because we have training features, training labels, testing features, testing labels. So essentially what we've created here is now a training set. We've split our data. Training set could also be referred to as training split yet another example of where machine learning has different names for different things. So set split same thing training split test split. This is what we've created. Remember,
4:44:48
the validation set is used often, but not always because our data set is quite simple. We're just sticking with the necessities training and test. But keep this in mind. One of your biggest, biggest, biggest hurdles in machine learning will be creating proper training and test sets. So it's a very important concept. With that being said, I did issue the challenge in the last video to visualize these numbers on a page. We haven't done that in this video. So let's move towards
4:45:17
that next. I'd like you to think of how could you make these more visual? Right. These are just numbers on a page right now. Maybe that plot lib can help. Let's find out. Hey, hey, hey, welcome back. In the last video, we split our data into training and test sets. And now later on, we're going to be building a model to learn patterns in the training data to relate to the testing data. But as I said, right now, our data is just numbers on a page. It's kind of
4:45:50
hard to understand. You might be able to understand this, but I prefer to get visual. So let's write this down. How might we better visualize our data? And I'm put a capital here. So we're grammatically correct. And this is where the data Explorers motto comes in. Visualize, visualize, visualize. Ha ha. Right. So if ever you don't understand a concept, one of the best ways to start understanding it more for me is to visualize it. So let's write a function to do just that.
4:46:29
We're going to call this plot predictions. We'll see why we call it this later on. That's the benefit of making these videos is that I've got a plan for the future. Although it might seem like I'm winging it, there is a little bit of behind the scenes happening here. So we'll have the train data, which is our X train. And then we'll have the train labels, which is our Y train. And we'll also have the test data. Yeah, that's a good idea. X test. And we'll also have the test
4:46:58
labels, equals Y test. Excuse me. I was looking at too many X's there. And then the predictions. And we'll set this to none, because we don't have any predictions yet. But as you might have guessed, we might have some later on. So we'll put a little doc string here, so that we're being nice and Pythonic. So plots training data, test data, and compares predictions. Nice and simple.
4:47:28
Nothing too outlandish. And then we're going to create a figure. This is where map plot lib comes in. Plot figure. And we'll go fig size equals 10, seven, which is my favorite hand in poker. And we'll plot the training data in blue also happens to be a good dimension for a map plot. Plot dot scatter. Train data. Creating a scatter plot here. We'll see what it does in a second. Color. We're going to give this a color of B for blue. That's what C stands for in map plot lib
4:48:00
scatter. We'll go size equals four and label equals training data. Now, where could you find information about this scatter function here? We've got command shift space. Is that going to give us a little bit of a doc string? Or sometimes if command not space is not working, you can also hover over this bracket. I think you can even hover over this.
4:48:26
There we go. But this is a little hard for me to read. Like it's there, but it's got a lot going on. X, Y, S, C, C map. I just like to go map plot lib scatter. There we go. We've got a whole bunch of information there. A little bit easier to read for me here. And then you can see some examples. Beautiful. So now let's jump back into here. So in our function plot predictions, we've taken some training data, test data. We've got the training data plotting in blue. What
4:49:03
color should we use for the testing data? How about green? I like that idea. Plot.scatter. Test data. Green's my favorite color. What's your favorite color? C equals G. You might be able to just plot it in your favorite color here. Just remember though, it'll be a little bit different from the videos. And then we're going to call this testing data. So just the exact same line is above, but with a different set of data. Now, let's check if there are predictions. So
4:49:33
are there predictions? So if predictions is not none, let's plot the predictions, plot the predictions, if they exist. So plot scatter test data. And why are we plotting the test data? Remember, what is our scatter function? Let's go back up to here. It takes in x and y. So our predictions are going to be compared to the testing data labels. So that's the whole game that we're playing here. We're going to train our model on the training data.
4:50:15
And then to evaluate it, we're going to get our model to predict the y values as with the input of x test. And then to evaluate our model, we compare how good our models predictions are. In other words, predictions versus the actual values of the test data set. But we're going to see this in practice. Rather than just talk about it. So let's do our predictions in red. And label equals predictions. Wonderful. So let's also show the legend, because, I mean,
4:50:55
we're legends. So we could just put in a mirror here. Now I'm kidding. Legend is going to show our labels on the map plot. So prop equals size and prop stands for properties. Well, it may or may not. I just like to think it does. That's how I remember it. So we have a beautiful function here to plot our data. Should we try it out? Remember, we've got hard coded inputs here, so we don't actually need to input anything to our function. We've got our train and test data
4:51:28
ready to go. If in doubt, run the code, let's check it out. Did we make a mistake in our plot predictions function? You might have caught it. Hey, there we go. Beautiful. So because we don't have any predictions, we get no red dots. But this is what we're trying to do. We've got a simple straight line. You can't get a much more simple data set than that. So we've got our training data in blue, and we've got our testing data in green. So the whole idea of what we're going to be doing
4:51:56
with our machine learning model is we don't actually really need to build a machine learning model for this. We could do other things, but machine learning is fun. So we're going to take in the blue dots. There's quite a pattern here, right? This is the relationship we have an x value here, and we have a y value. So we're going to build a model to try and learn the pattern of these blue dots, so that if we fed our model, our model, the x values of the green dots,
4:52:25
could it predict the appropriate y values for that? Because remember, these are the test data set. So pass our model x test to predict y test. So blue dots as input, green dots as the ideal output. This is the ideal output, a perfect model would have red dots over the top of the green dots. So that's what we will try to work towards. Now, we know the relationship between x and y. How do we know that? Well, we set that up above here. This is our weight and bias.
4:52:53
We created that line y equals weight times x plus bias, which is the simple version of the linear regression formula. So mx plus c, you might have heard that in high school algebra, so gradient plus intercept. That's what we've got. With that being said, let's move on to the next video and build a model. Well, this is exciting. I'll see you there. Welcome back. In the last video, we saw how to get visual with our data. We followed the data
4:53:24
explorer's motto of visualize, visualize, visualize. And we've got an idea of the training data that we're working with and the testing data that we're trying to build a model to learn the patterns in the training data, essentially this upwards trend here, to be able to predict the testing data. So I just want to give you another heads up. I took a little break after the recording last video. And so now my colab notebook has disconnected. So I'm going to click reconnect.
4:53:55
And my variables here may not work. So this is what might happen on your end. If you take a break from using Google Colab and come back, if I try to run this function, they might have been saved, it looks like they have. But if not, you can go restart and run all. This is typically one of the most helpful troubleshooting steps of using Google Colab. If a cell, say down here isn't working, you can always rerun the cells above. And that may help with a lower cell here, such as if this
4:54:32
function wasn't instantiated because this cell wasn't run, and we couldn't run this cell here, which calls this function here, we just have to rerun this cell above so that we can run this one. But now let's get into building our first PyTorch model. We're going to jump straight into the code. So our first PyTorch model. Now this is very exciting. Let's do it. So we'll turn this into Markdown. Now we're going to create a linear regression model.
4:55:09
So look at linear regression formula again, we're going to create a model that's essentially going to run this computation. So we need to create a model that has a parameter for A, a parameter for B, and in our case it's going to be weight and bias, and a way to do this forward computation. What I mean by that, we're going to see with code. So let's do it. We'll do it with pure PyTorch. So create a linear regression model class. Now if you're not experienced with using Python classes,
4:55:44
I'm going to be using them throughout the course, and I'm going to call this one linear regression model. If you haven't dealt with Python classes before, that's okay. I'm going to be explaining what we're doing as we're doing it. But if you'd like a deeper dive, I'd recommend you to real Python
4:56:04
classes. OOP in Python three. That's a good rhyming. So I'm just going to link this here. Because we're going to be building classes throughout the course, I'd recommend getting familiar with OOP, which is object oriented programming, a little bit of a mouthful, hence the OOP in Python. To do so, you can use the following resource from real Python. But when I'm going to go through that now, I'd rather just code it out and talk it out while we
4:56:48
do it. So we've got a class here. Now the first thing you might notice is that the class inherits from nn.module. And you might be wondering, well, what's nn.module? Well, let's write down here, almost everything in PyTorch inherits from nn.module. So you can imagine nn.module as the Lego building bricks of PyTorch model. And so nn.module has a lot of helpful inbuilt things that's going to help us build our PyTorch models. And of course, how could you learn more about it?
4:57:25
Well, you could go nn.module, PyTorch. Module. Here we go. Base class for all neural network modules. Wonderful. Your models should also subclass this class. So that's what we're building. We're building our own PyTorch model. And so the documentation here says that your models should also subclass this class. And another thing with PyTorch, this is what makes it, it might seem very confusing when you first begin. But modules can contain other modules. So what I mean by being a
4:57:56
Lego brick is that you can stack these modules on top of each other and make progressively more complex neural networks as you go. But we'll leave that for later on. For now, we're going to start with something nice and simple. And let's clean up our web browser. So we're going to create a constructor here, which is with the init function. It's going to take self as a parameter. If you're not sure of what's going on here, just follow along with the code for now. And I'd encourage you
4:58:29
to read this documentation here after the video. So then we have super dot init. I know when I first started learning this, I was like, why do we have to write a knit twice? And then what's super and all that jazz. But just for now, just take this as being some required Python syntax. And then we have self dot weights. So that means we're going to create a weights parameter. We'll see why we do this in a second. And to create that parameter, we're going to use nn dot parameter.
4:59:00
And just a quick reminder that we imported nn from torch before. And if you remember, nn is the building block layer for neural networks. And within nn, so nn stands for neural network is module. So we've got nn dot parameter. Now, we're going to start with random parameters. So torch dot rand n. One, we're going to talk through each of these in a second. So I'm also going to put requires, requires grad equals true. We haven't touched any of these, but that's okay.
4:59:40
D type equals torch dot float. So let's see what nn parameter tells us. What do we have here?
4:59:53
A kind of tensor that is to be considered a module parameter. So we've just created a module using nn module. Parameters are torch tensor subclasses. So this is a tensor in itself that have a very special property when used with modules. When they're assigned as a module attribute, they are automatically added to the list of its parameters. And we'll appear e g in module dot parameters iterator. Oh, we're going to see that later on. Assigning a tensor
5:00:20
doesn't have such effect. So we're creating a parameter here. Now requires grad. What does that mean? Well, let's just rather than just try to read the doc string collab, let's look it up. nn dot parameter. What does it say requires grad optional. If the parameter requires gradient. Hmm. What does requires gradient mean? Well, let's come back to that in a second. And then for now, I just want you to think about it. D type equals torch dot float. Now,
5:00:56
the data type here torch dot float is, as we've discussed before, is the default for pytorch to watch dot float. This could also be torch dot float 32. So we're just going to leave it as torch float 32, because pytorch likes to work with flight 32. Now, do we have
5:01:17
this by default? We do. So we don't necessarily have to set requires grad equals true. So just keep that in mind. So now we've created a parameter for the weights. We also have to create a parameter for the bias. Let's finish creating this. And then we'll write the code, then we'll talk about it. So rand n. Now requires grad equals true. And d type equals torch dot float. There we go. And now we're going to write a forward method. So forward method to define the computation
5:02:02
in the model. So let's go def forward, which self takes in a parameter x, which is data, which X is expected to be of type torch tensor. And it returns a torch dot tensor. And then we go here. And so we say X, we don't necessarily need this comment. I'm just going to write it anyway. X is the input data. So in our case, it might be the training data. And then from here, we want it to return self dot weights times X plus self dot bias. Now, where have we seen this before?
5:02:47
Well, this is the linear regression formula. Now, let's take a step back into how we created our data. And then we'll go back through and talk a little bit more about what's going on here. So if we go back up to our data, where did we create that? We created it here. So you see how we've created known parameters, weight and bias. And then we created our y variable, our target, using the linear regression formula, wait times X plus bias, and X were a range of numbers.
5:03:23
So what we've done with our linear regression model that we've created from scratch, if we go down here, we've created a parameter, weights. This could just be weight, if we wanted to. We've created a parameter here. So when we created our data, we knew what the parameters weight and bias were. The whole goal of our model is to start with random numbers. So these are going to be random parameters. And to look at the data, which in our case will be the training samples,
5:03:59
and update those random numbers to represent the pattern here. So ideally, our model, if it's learning correctly, will take our weight, which is going to be a random value, and our bias, which is going to be a random value. And it will run it through this forward calculation, which is the same formula that we use to create our data. And it will adjust the weight and bias to represent as close as possible, if not perfect, the known parameters. So that's the premise of
5:04:34
machine learning. And how does it do this? Through an algorithm called gradient descent. So I'm just going to write this down because we've talked a lot about this, but I'd like to just tie it together here. So what our model does, so start with random values, weight and bias, look at training data, and adjust the random values to better represent the, or get closer to the ideal values. So the weight and bias values we use to create the data. So that's what it's going to do. It's going to
5:05:33
start with random values, and then continually look at our training data to see if it can adjust those random values to be what would represent this straight line here. Now, how does it do so? How does it do so? Through two main algorithms. So one is gradient descent, and two is back propagation. So I'm going to leave it here for the time being, but we're going to continue talking about this gradient descent is why we have requires grad equals true. And so what this is going to
5:06:21
do is when we run computations using this model here, pytorch is going to keep track of the gradients of our weights parameter and our bias parameter. And then it's going to update them through a combination of gradient descent and back propagation. Now, I'm going to leave this as extracurricular for you to look through and gradient descent and back propagation. I'm going to add some resources here. There will also be plenty of resources in the pytorch workflow fundamentals
5:06:51
book chapter on how these algorithms work behind the scenes. We're going to be focused on the code, the pytorch code, to trigger these algorithms behind the scenes. So pytorch, lucky for us, has implemented gradient descent and back propagation for us. So we're writing the higher level code here to trigger these two algorithms. So in the next video, we're going to step through this a little bit more, and then further discuss some of the most useful and required modules of pytorch,
5:07:21
particularly an N and a couple of others. So let's leave it there, and I'll see you in the next video. Welcome back. In the last video, we covered a whole bunch in creating our first pytorch model that inherits from nn.module. We talked about object oriented programming and how a lot of pytorch uses object oriented programming. I can't say that. I might just say OOP for now. What I've done since last video, though, is I've added two resources here for gradient descent
5:07:51
and back propagation. These are two of my favorite videos on YouTube by the channel three blue one brown. So this is on gradient descent. I would highly recommend watching this entire series, by the way. So that's your extra curriculum for this video, in particular, and for this course overall is to go through these two videos. Even if you're not sure entirely what's happening, you will gain an intuition for the code that we're going to be writing with pytorch.
5:08:17
So just keep that in mind as we go forward, a lot of what pytorch is doing behind the scenes for us is taking care of these two algorithms for us. And we also created two parameters here in our model where we've instantiated them as random values. So one parameter for each of the ones that we use, the weight and bias for our data set. And now I want you to keep in mind that we're working with a simple data set here. So we've created our known parameters. But in a data set that you
5:08:50
haven't created by yourself, you've maybe gathered that from the internet, such as images, you won't be necessarily defining these parameters. Instead, another module from nn will define the parameters for you. And we'll work out what those parameters should end up being. But since we're working with a simple data set, we can define our two parameters that we're trying to estimate. This is a key point here is that our model is going to start with random values. That's the
5:09:21
annotation I've added here. Start with a random weight value using torch random. And then we've told it that it can update via gradient descent. So pytorch is going to track the gradients of this parameter for us. And then we've told it that the D type we want is float 32. We don't necessarily need these two set explicitly, because a lot of the time the default in pytorch is to set these two requires grad equals true and d type equals torch dot float. Does that for us
5:09:49
behind the scenes? But just to keep things as fundamental and as straightforward as possible, we've set all of this explicitly. So let's jump into the keynote. I'd just like to explain what's going on one more time in a visual sense. So here's the exact code that we've just written. I've just copied it from here. And I've just made it a little bit more colorful. But here's what's going on. So when you build a model in pytorch, it subclasses the nn.modgable
5:10:21
class. This contains all the building blocks for neural networks. So our class of model, subclasses nn.modgable. Now, inside the constructor, we initialize the model parameters. Now, as we'll see, later on with bigger models, we won't necessarily always explicitly create the weights and biases. We might initialize whole layers. Now, this is a concept we haven't touched on yet, but we might initialize a list of layers or whatever we need. So basically, what happens in here is that
5:10:57
we create whatever variables that we need for our model to use. And so these could be different layers from torch.nn, single parameters, which is what we've done in our case, hard coded values, or even functions. Now, we've explicitly set requires grad equals true for our model parameters. So this, in turn, means that pytorch behind the scenes will track all of the gradients for these parameters here for use with torch.auto grad. So torch.auto grad module of pytorch is what
5:11:31
implements gradient descent. Now, a lot of this will happen behind the scenes for when we write our pytorch training code. So if you'd like to know what's happening behind the scenes, I'd highly recommend you checking out these two videos, hence is why I've linked them here.
5:11:46
Oh, and for many pytorch.nn modules requires grad is true is set by default.
5:11:53
Finally, we've got a forward method. Now, any subclass of nn.modgable, which is what we've done, requires a forward method. Now, we can see this in the documentation. If we go torch dot nn.modgable.
5:12:10
Click on module. Do we have forward?
5:12:16
Yeah, there we go. So forward, we've got a lot of things built into an nn.modgable. So you see here, this is a subclass of an nn.modgable. And then we have forward. So forward is what defines the computation performed at every call. So if we were to call linear regression model and put some data through it, the forward method is the operation that this module does that this model does. And in our case, our forward method is
5:12:46
the linear regression function. So keep this in mind, any subclass of nn.modgable needs to override the forward method. So you need to define a forward method if you're going to subclass nn.modgable. We'll see this very hands on. But for now, I believe that's enough coverage of what we've done. If you have any questions, remember, you can ask it in the discussions. We've got a fair bit going on here. But I think we've broken it down a fair bit. The next step is for us to,
5:13:17
I know I mentioned this in a previous video is to cover some PyTorch model building essentials. But we're going to cover a few more of them. We've seen some already. But the next way to really start to understand what's going on is to check the contents of our model, train one, and make some predictions with it. So let's get hands on with that in the next few videos. I'll see you there.
5:13:42
Welcome back. In the last couple of videos, we stepped through creating our first PyTorch model. And it looks like there's a fair bit going on here. But some of the main takeaways is that almost every model in PyTorch inherits from nn.modgable. And if you are going to inherit from nn.modgable, you should override the forward method to define what computation is happening in your model. And for later on, when our model is learning things, in other words, updating its weights and
5:14:10
bias values from random values to values that better fit the data, it's going to do so via gradient descent and back propagation. And so these two videos are some extra curriculum for what's happening behind the scenes. But we haven't actually written any code yet to trigger these two. So I'll refer back to these when we actually do write code to do that. For now, we've just got a model that defines some forward computation. But speaking of models, let's have
5:14:41
a look at a couple of PyTorch model building essentials. So we're not going to write too much code for this video, and it's going to be relatively short. But I just want to introduce you to some of the main classes that you're going to be interacting with in PyTorch. And we've seen some of these already. So one of the first is torch.nn. So contains all of the building blocks for computational graphs. Computational graphs is another word for neural networks.
5:15:09
Well, actually computational graphs is quite general. I'll just write here, a neural network can be considered a computational graph. So then we have torch.nn.parameter. We've seen this. So what parameters should our model try and learn? And then we can write here often a PyTorch layer from torch.nn will set these for us. And then we've got torch.nn.module, which is what we've seen here. And so torch.nn.module is the base class for all neural network modules.
5:16:03
If you subclass it, you should overwrite forward, which is what we've done here. We've created our own forward method. So what else should we cover here? We're going to see these later on, but I'm going to put it here, torch.optim. This is where the optimizers in PyTorch live. They will help with gradient descent. So optimizer, an optimizer is, as we've said before, that our model starts with random values. And it looks at training data and adjusts the random
5:16:44
values to better represent the ideal values. The optimizer contains algorithm that's going to optimize these values, instead of being random, to being values that better represent our data. So those algorithms live in torch.optim. And then one more for now, I'll link to extra resources. And we're going to cover them as we go. That's how I like to do things, cover them as we need them. So all nn.module. So this is the forward method. I'm just going to explicitly say here that all
5:17:19
nn.module subclasses require you to overwrite forward. This method defines what happens in the forward computation. So in our case, if we were to pass some data to our linear regression model, the forward method would take that data and perform this computation here. And as your models get bigger and bigger, ours is quite straightforward here. This forward computation can be as simple or as complex as you like, depending on what you'd
5:17:54
like your model to do. And so I've got a nice and fancy slide here, which basically reiterates what we've just discussed. PyTorch is central neural network building modules. So the module torch.nn, torch.nn.module, torch.optim, torch.utils.dataset. We haven't actually talked about this yet. And I believe there's one more data loader. We're going to see these two later on. But these are very helpful when you've got a bit more of a complicated data set. In our case,
5:18:27
we've got just 50 integers for our data set. We've got a simple straight line. But when we need to create more complex data sets, we're going to use these. So this will help us build models. This will help us optimize our models parameters. And this will help us load data. And if you'd like more, one of my favorite resources is the PyTorch cheat sheet. Again, we're referring back to the documentation. See, all of this documentation, right? As I said, this course is
5:18:56
not a replacement for the documentation. It's just my interpretation of how one should best become familiar with PyTorch. So we've got imports, the general import torch from torch.utils.dataset data loader. Oh, did you look at that? We've got that mentioned here, data, data set data loader. And torch, script and jit, neural network API. I want an X. I'll let you go through here. We're covering some of the most fundamental ones here. But there's, of course, PyTorch is
5:19:26
quite a big library. So some extra curricula for this video would be to go through this for five to 10 minutes and just read. You don't have to understand them all. We're going to start to get more familiar with all of these. We're not all of them because, I mean, that would require making videos for the whole documentation. But a lot of these through writing them via code.
5:19:47
So that's enough for this video. I'll link this PyTorch cheat sheet in the video here. And in the next video, how about we, we haven't actually checked out what happens if we do create an instance of our linear regression model. I think we should do that. I'll see you there.
5:20:09
Welcome back. In the last video, we covered some of the PyTorch model building essentials. And look, I linked a cheat sheet here. There's a lot going on. There's a lot of text going on in the page. Of course, the reference material for here is in the Learn PyTorch book. PyTorch model building essentials under 0.1, which is the notebook we're working on here. But I couldn't help myself. I wanted to add some color to this. So before we inspect our model, let's just add a little bit
5:20:37
of color to our text on the page. We go to whoa. Here's our workflow. This is what we're covering in this video, right? Or in this module, 0.1. But to get data ready, here are some of the most important PyTorch modules. Torchvision.transforms. We'll see that when we cover computer vision later on. Torch.utils.data.data set. So that's if we want to create a data set that's a little bit more complicated than because our data set is so simple, we haven't used either of these
5:21:05
data set creator or data loader. And if we go build a picker model, well, we can use torch.nn. We've seen that one. We've seen torch.nn.module. So in our case, we're building a model. But if we wanted a pre-trained model, well, there's some computer vision models that have already been built for us in torchvision.models. Now torchvision stands for PyTorch's computer vision module. So we haven't covered that either. But this is just a spoiler for what's coming on
5:21:34
later on. Then if the optimizer, if we wanted to optimize our model's parameters to better represent a data set, we can go to torch.optim. Then if we wanted to evaluate the model, well, we've got torch metrics for that. We haven't seen that, but we're going to be hands-on with all of these later on. Then if we wanted to improve through experimentation, we've got torch.utils.tensorboard. Hmm. What's this? But again, if you want more,
5:22:00
there's some at the PyTorch cheat sheet. But now this is just adding a little bit of color and a little bit of code to our PyTorch workflow. And with that being said, let's get a little bit deeper into what we've built, which is our first PyTorch model. So checking the contents of our PyTorch model. So now we've created a model. Let's see what's inside. You might already be able
5:22:30
to guess this by the fact of what we've created in the constructor here in the init function. So what do you think we have inside our model? And how do you think we'd look in that? Now, of course, these are questions you might not have the answer to because you've just, you're like, Daniel, I'm just starting to learn PyTorch. I don't know these, but I'm asking you just to start thinking about these different things, you know? So we can check out our model parameters or what's
5:22:59
inside our model using, wait for it, dot parameters. Oh, don't you love it when things are nice and simple? Well, let's check it out. Hey, well, first things we're going to do is let's create a random seed. Now, why are we creating a random seed? Well, because recall, we're creating these parameters with random values. And if we were to create them with outer random seed, we would get different values every time. So for the sake of the educational sense, for the sake of this video,
5:23:38
we're going to create a manual seed here, torch dot manual seed. I'm going to use 42 or maybe 43, I could use 43 now 42 because I love 42. It's the answer to the universe. And we're going to create an instance of the model that we created. So this is a subclass of an end up module.
5:24:02
So let's do it. Model zero, because it's going to be the zeroth model, the first model that we've ever created in this whole course, how amazing linear regression model, which is what our class is called. So we can just call it like that. That's all I'm doing, just calling this class. And so let's just see what happens there. And then if we go model zero, what does it give us? Oh, linear regression. Okay, it doesn't give us much. But we want to find out what's going on in here.
5:24:32
So check out the parameters. So model zero dot parameters. What do we get from this? Oh, a generator. Well, let's turn this into a list that'll be better to look at. There we go. Oh, how exciting is that? So parameter containing. Look at the values tensor requires grad equals true parameter containing wonderful. So these are our model parameters. So why are they the values that they are? Well, it's because we've used torch rand n. Let's see what happens if we go, let's just create torch dot
5:25:20
rand n one, what happens? We get a value like that. And now if we run this again,
5:25:28
we get the same values. But if we run this again, so keep this in one two, three, four, five, actually, that's, wow, that's pretty cool that we got a random value that was all in order, four in a row. Can we do it twice in a row? Probably not. Oh, we get it the same one. Now, why is that? Oh, we get a different one. Did we just get the same one twice? Oh, my gosh, we got the same value twice in a row. You saw that. You saw that. That's incredible. Now,
5:25:55
the reason why we get this is because this one is different every time because there's no random seed. Watch if we put the random seed here, torch dot manual seed, 42, 3, 3, 6, 7, what happens?
5:26:13
3, 3, 6, 7, what happens? 3, 3, 6, 7. Okay. And what if we commented out the random seed here, initialized our model, different values, two, three, five, two, three, four, five, it must like that value. Oh, my goodness. Let me know if you get that value, right? So if we keep going, we get different values every single time. Why is this? Why are we getting different values every single time? You might be, Daniel, you sound like a broken record, but I'm trying to
5:26:43
really drive home the fact that we initialize our models with random parameters. So this is the essence of what our machine learning models and deep learning models are going to do. Start with random values, weights and bias. Maybe we've only got two parameters here, but the future models that we build might have thousands. And so of course, we're not going to do them all by hand. We'll see how we do that later on. But for now, we start with random values. And our ideal model
5:27:08
will look at the training data and adjust these random values. But just so that we can get reproducible results, I'll get rid of this cell. I've set the random seed here. So you should be getting similar values to this. If you're not, because there's maybe some sort of pytorch update and how the random seeds calculated, you might get slightly different values. But for now, we'll use torch.manualc.42. And I want you to just be aware of this can be a little bit confusing.
5:27:37
If you just do the list of parameters, for me, I understand it better if I list the name parameters. So the way we do that is with model zero, and we call state dict on it. This is going to give us our dictionary of the parameters of our model. So as you can see here, we've got weights, and we've got bias, and they are random values. So where did weights and bias come from? Well, of course, they came from here, weights, bias. But of course, as well up here,
5:28:12
we've got known parameters. So now our whole goal is what? Our whole goal is to build code, or write code, that is going to allow our model to look at these blue dots here, and adjust this weight and bias value to be weights as close as possible to weight and bias. Now, how do we go from here and here to here and here? Well, we're going to see that in future videos, but the closer we get these values to these two, the better we're going to be able to
5:28:58
predict and model our data. Now, this principle, I cannot stress enough, is the fundamental entire foundation, the fundamental foundation. Well, good description, Daniel. The entire foundation of deep learning, we start with some random values, and we use gradient descent and back propagation, plus whatever data that we're working with to move these random values as close as possible to the ideal values. And in most cases, you won't know what the ideal values are.
5:29:30
But in our simple case, we already know what the ideal values are. So just keep that in mind going forward. The premise of deep learning is to start with random values and make them more representative closer to the ideal values. With that being said, let's try and make some predictions with our model as it is. I mean, it's got random values. How do you think the predictions will go? So I think in the next video, we'll make some predictions
5:29:55
on this test data and see what they look like. I'll see you there. Welcome back. In the last video, we checked out the internals of our first PyTorch model. And we found out that because we're creating our model with torch dot or the parameters of our model, with torch dot rand, they begin as random variables. And we also discussed the entire premise of deep learning is to start with random numbers and slowly progress those towards more ideal numbers, slightly less random numbers based
5:30:28
on the data. So let's see, before we start to improve these numbers, let's see what their predictive power is like right now. Now you might be able to guess how well these random numbers will be able to predict on our data. You're not sure what that predicting means? Let's have a look. So making predictions using torch dot inference mode, something we haven't seen. But as always, we're going to discuss it while we use it. So to check our models predictive power, let's see how well
5:31:07
it predicts Y test based on X test. Because remember again, another premise of a machine learning model is to take some features as input and make some predictions close to some sort of labels. So when we pass data through our model, it's going to run it through the forward method.
5:31:41
So here's where it's a little bit confusing. We defined a forward method and it takes X as input. Now I've done a little X, but we're going to pass it in a large X as its input. But the reason why I've done a little X is because oftentimes in pytorch code, you're going to find all over the internet is that X is quite common, commonly used in the forward method here, like this as the input data. So I've just left it there because that's what you're going to find quite often.
5:32:07
So let's test it out. We haven't discussed what inference mode does yet, but we will make predictions with model. So with torch dot inference mode, let's use it. And then we will discuss what's going on. Y threads equals a model zero X test. So that's all we're doing. We're passing the X test data
5:32:34
through our model. Now, when we pass this X test in here, let's remind ourselves of what X test is. X test 10 variables here. And we're trying to our ideal model will predict the exact values of Y test. So this is what our model will do if it's a perfect model. It will take these X test values as input, and it will return these Y test values as output. That's an ideal model. So the predictions are the
5:33:06
exact same as the test data set. How do you think our model will go considering it's starting with random values as its parameters? Well, let's find out, hey. So what can that Y threads?
5:33:20
Oh, what's happened here? Not implemented error. Ah, this is an error I get quite often in Google Colab when I'm creating a high-torch model. Now, it usually happens. I'm glad we've stumbled upon this. And I think I know the fix. But if not, we might see a little bit of troubleshooting in this video is that when we create this, if you see this not implemented error, right, it's saying that the Ford method. Here we go. Ford implemented. There we go. It's a little bit of a rabbit hole
5:33:52
this not implemented area. I've come across it a fair few times and it took me a while to figure out that for some reason the spacing. So in Python, you know how you have space space and that defines a function space space. There's another thing there and another line there. For some reason, if you look at this line in my notebook, and by the way, if you don't have these lines or if you don't have these numbers, you can go into tools, settings, editor, and then you can define them here.
5:34:19
So show line numbers, show notation guides, all that sort of jazz there. You can customize what's going on. But I just have these two on because I've run into this error a fair few times. And so it's because that this Ford method is not in line with this bracket here. So we need to highlight this and click shift tab, move it over. So now you see that it's in line here. And then if we run this, won't change any output there. See, that's the hidden gotcha. Is that when we ran this before,
5:34:51
it found no error. But then when we run it down here, it works. So just keep that in mind. I'm really glad we stumbled upon that because indentation errors, not implemented errors, one of the most common errors you'll find in PyTorch, or in, well, when you're writing PyTorch code in Google CoLab, I'm not sure why, but it just happens. So these are our models predictions so far by running the test data through our models Ford method that we defined. And so if
5:35:24
we look at Y test, are these close? Oh my gosh, they are shocking. So why don't we visualize them?
5:35:33
Plot predictions. And we're going to put in predictions equals Y threads.
5:35:40
Let's have a look. Oh my goodness. All the way over here. Remember how we discussed before that an ideal model will have, what, red dots on top of the green dots because our ideal model will be perfectly predicting the test data. So right now, because our model is initialized with random parameters, it's basically making random predictions. So they're extremely far from where our ideal predictions are, is that we'll have some training data. And our model predictions,
5:36:09
when we first create our model will be quite bad. But we want to write some code that will hopefully move these red dots closer to these green dots. I'm going to see how we can do that in later videos. But we did one thing up here, which we haven't discussed, which is with torch dot inference mode. Now this is a context manager, which is what happens when we're making predictions. So making predictions, another word for predictions is inference torch uses inference. So I'll try
5:36:38
to use that a bit more, but I like to use predictions as well. We could also just go Y preds equals model zero dot X test. And we're going to get quite a similar output.
5:36:56
Right. But I've put on inference mode because I want to start making that a habit for later on, when we make predictions, put on inference mode. Now why do this? You might notice something different. What's the difference here between the outputs? Y preds equals model. There's no inference mode here, no context manager. Do you notice that there's a grad function here? And we don't need to go into discussing what exactly this is doing here. But do you notice that this one is lacking that
5:37:24
grad function? So do you remember how behind the scenes I said that pie torch does a few things
5:37:31
with requires grad equals true, it keeps track of the gradients of different parameters so that they can be used in gradient descent and back propagation. Now what inference mode does is it turns off that gradient tracking. So it essentially removes all of the, because when we're doing inference, we're not doing training. So we don't need to keep track of the gradient. So we don't need to do to keep track of how we should update our models. So inference mode disables all of the
5:38:03
useful things that are available during training. What's the benefit of this? Well, it means that pie torch behind the scenes is keeping track of less data. So in turn, it will, with our small data set, it probably won't be too dramatic. But with a larger data set, it means that your predictions will potentially be a lot faster because a whole bunch of numbers aren't being kept track of or a whole bunch of things that you don't need during prediction mode or inference
5:38:31
mode. That's why it's called inference mode. I'm not being saved to memory. If you'd like to learn more about this, you go pie torch inference mode Twitter. I just remember to search for Twitter because they did a big tweet storm about it. Here we go. So oh, this is another thing that we can cover. I'm going to copy this in here. But there's also a blog post about what's going on behind the scenes. Long story short, it makes your code faster. Want to make your inference code and pie
5:39:03
torch run faster? Here's a quick thread on doing exactly that. And that's what we're doing. So I'm going to write down here. See more on inference mode here.
5:39:17
And I just want to highlight something as well is that they referenced torch no grad with the torch inference mode context manager. Inference mode is fairly new in pie torch. So you might see a lot of code existing pie torch code with torch dot no grad. You can use this as well. Why? Preds equals model zero. And this will do much of the same as what inference mode is doing. But inference mode has a few things that are advantages over no grad, which are discussed in
5:39:45
this thread here. But if we do this, we get very similar output to what we got before. Grad function. But as you'll read in here and in the pie torch documentation, inference mode is the favored way of doing inference for now. I just wanted to highlight this. So you can also do something similar with torch dot no grad. However, inference mode is preferred. Alrighty. So I'm just going to comment this out. So we just have one thing going on there. The main takeaway
5:40:32
from this video is that when we're making predictions, we use the context manager torch dot inference mode. And right now, because our models variables or internal parameters are randomly initialized, our models predictions are as good as random. So they're actually not too far off where our values are. At least the red dots aren't like scattered all over here. But in the upcoming videos, we're going to be writing some pie torch training code to move these values
5:41:04
closer to the green dots by looking at the training data here. So with that being said, I'll see you in the next video. Friends, welcome back. In the last video, we saw that our model performs pretty poorly. Like, ideally, these red dots should be in line with these green dots. And we know that because why? Well, it's because our model is initialized with random parameters. And I just want to put a little note here. You don't necessarily have to initialize your model
5:41:38
with random parameters. You could initialize it with this could be zero. Yeah, these two values, weights can bias could be zero and you could go from there. Or you could also use the parameters from another model. But we're going to see that later on. That's something called transfer learning. That's just a little spoiler for what's to come. And so we've also discussed that an ideal model will replicate these known parameters. So in other words, start with random unknown parameters,
5:42:09
these two values here. And then we want to write some code for our model to move towards estimating the ideal parameters here. Now, I just want to be explicit here and write down some intuition before we jump into the training code. But this is very exciting. We're about to get into training our very first machine learning model. So what's right here, the whole idea of training is for a model to move from some unknown parameters, these may be random to some known parameters.
5:42:49
Or in other words, from a poor representation, representation of the data to a better representation of the data. And so in our case, would you say that our models representation of the green dots here with this red dots, is that a good representation? Or is that a poor representation? I mean, I don't know about you, but I would say that to me, this is a fairly poor representation. And one way to measure the representation between your models outputs, in our case, the red dots,
5:43:28
the predictions, and the testing data, is to use a loss function. So I'm going to write this down here. This is what we're moving towards. We're moving towards training, but we need a way to measure how poorly our models predictions are doing. So one way to measure how poor or how wrong your models predictions are, is to use a loss function. And so if we go pytorch loss functions, we're going to see that pytorch has a fair few loss functions built in. But the essence
5:44:06
of all of them is quite similar. So just wait for this to load my internet's going a little bit slow today, but that's okay. We're not in a rush here. We're learning something fun. If I search here for loss, loss functions, here we go. So yeah, this is torch in N. These are the basic building blocks for graphs, whole bunch of good stuff in here, including loss functions. Beautiful. And this is another thing to note as well, another one of those scenarios where
5:44:29
there's more words for the same thing. You might also see a loss function referred to as a criterion. There's another word called cost function. So I might just write this down so you're aware of it. Yeah, cost function versus loss function. And maybe some formal definitions about what all of these are. Maybe they're used in different fields. But in the case of we're focused on machine learning, right? So I'm just going to go note, loss function may also be called cost function or criterion in
5:45:03
different areas. For our case, we're going to refer to it as a loss function. And let's just formally define a loss function here, because we're going to go through a fair few steps in the upcoming videos. So this is a warning, nothing we can't handle. But I want to put some formal definitions on things. We're going to see them in practice. That's what I prefer to do, rather than just sit here defining stuff. This lecture has already had enough text on the page.
5:45:30
So hurry up and get into coding Daniel. A loss function is a function to measure how wrong your models predictions are to the ideal outputs. So lower is better. So ideally, think of a measurement, how could we measure the difference between the red dots and the green dots? One of the simplest ways to do so would be just measure the distance here, right? So if we go, let's just estimate this is 035 to 0.8. They're abouts. So what's the difference there? About 0.45.
5:46:09
Then we could do the same again for all of these other dots, and then maybe take the average of that.
5:46:15
Now, if you've worked with loss functions before, you might have realized that I've just reproduced mean absolute error. But we're going to get to that in a minute. So we need a loss function. I'm going to write down another little dot point here. This is just setting up intuition. Things we need to train. We need a loss function. This is PyTorch. And this is machine learning in general, actually. But we're focused on PyTorch. We need an optimizer. What does the optimizer do?
5:46:43
Takes into account the loss of a model and adjusts the model's parameters. So the parameters recall our weight and bias values. Weight and biases. We can check those or bias. We can check those by going model dot parameter or parameters. But I also like, oh, that's going to give us a generator, isn't it? Why do we not define the model yet? What do we call our model? Oh, model zero. Excuse me. I forgot where. I'm going to build a lot of models in this course. So we're giving them numbers.
5:47:24
Modeled up parameters. Yeah, we've got a generator. So we'll turn that into a list. But model zero, if we want to get them labeled, we want state dict here.
5:47:35
There we go. So our weight is this value. That's a random value we've set. And there's the bias. And now we've only got two parameters for our model. So it's quite simple. However, the principles that we're learning here are going to be the same principles, taking a loss function, trying to minimize it, so getting it to lower. So the ideal model will predict exactly what our test data is. And an optimizer will take into account the loss and will adjust a model's parameter.
5:48:02
And our case weights and bias to be, let's finish this definition takes into account the loss of a model and adjust the model's parameters, e.g. weight and bias, in our case, to improve the loss function. And specifically, for PyTorch, we need a training loop and a testing loop. Now, this is what we're going to work towards building throughout the next couple of videos. We're going to focus on these two first, the loss function and optimizer. There's the formal
5:48:44
definition of those. You're going to find many different definitions. That's how I'm going to find them. Loss function measures how wrong your model's predictions are, lower is better, optimizer takes into account the loss of your model. So how wrong it is, and starts to move these two values into a way that improves where these red dots end up. But these, again, these principles of a loss function and an optimizer can be for models with two parameters or models
5:49:11
with millions of parameters, can be for computer vision models, or could be for simple models like ours that predict the dots on a straight line. So with that being said, let's jump into the next video. We'll start to look a little deeper into loss function, row problem, and an optimizer. I'll see you there. Welcome back. We're in the exciting streak of videos coming up here. I mean, the whole course is fun. Trust me. But this is really exciting because training your first machine
5:49:40
learning model seems a little bit like magic, but it's even more fun when you're writing the code yourself what's going on behind the scenes. So we discussed that the whole concept of training is from going unknown parameters, random parameters, such as what we've got so far to parameters that better represent the data. And we spoke of the concept of a loss function. We want to minimize the loss function. That is the whole idea of a training loop in PyTorch,
5:50:04
or an optimization loop in PyTorch. And an optimizer is one of those ways that can nudge the parameters of our model. In our case, weights or bias towards values rather than just being random values like they are now towards values that lower the loss function. And if we lower the loss function, what does a loss function do? It measures how wrong our models predictions are compared to the ideal outputs. So if we lower that, well, hopefully we move
5:50:34
these red dots towards the green dots. And so as you might have guessed, PyTorch has some built in functionality for implementing loss functions and optimizers. And by the way, what we're covering so far is in the train model section of the PyTorch workflow fundamentals, I've got a little nice table here, which describes a loss function. What does it do? Where does it live in PyTorch? Common values, we're going to see some of these hands on. If you'd like to read about it,
5:51:02
of course, you have the book version of the course here. So loss functions in PyTorch, I'm just in docstorch.nn. Look at this. Look at all these loss functions. There's far too many for us to go through all in one hit. So we're just going to focus on some of the most common ones. Look at that. We've got about what's our 15 loss functions, something like that? Well, truth be told is that which one should use? You're not really going to know unless you start to work hands
5:51:28
on with different problems. And so in our case, we're going to be looking at L1 loss. And this is an again, once more another instance where different machine learning libraries have different names for the same thing, this is mean absolute error, which we kind of discussed in the last video, which is if we took the distance from this red dot to this green dot and say at 0.4, they're about 0.4, 0.4, and then took the mean, well, we've got the mean absolute error. But in PyTorch,
5:51:58
they call it L1 loss, which is a little bit confusing because then we go to MSE loss, which is mean squared error, which is L2. So naming conventions just takes a little bit of getting used to this is a warning for you. So let's have a look at the L1 loss function. Again, I'm just making you aware of where the other loss functions are. We'll do with some binary cross entropy loss later in the course. And maybe even is that categorical cross entropy?
5:52:26
We'll see that later on. But all the others will be problem specific. For now, a couple of loss functions like this, L1 loss, MSE loss, we use for regression problems. So that's predicting a number. Cross entropy loss is a loss that you use with classification problems. But we'll see those hands on later on. Let's have a look at L1 loss. So L1 loss creates a criterion. As I said, you might hear the word criterion used in PyTorch for a loss function. I typically call them loss functions.
5:52:55
The literature typically calls it loss functions. That measures the mean absolute error. There we go. L1 loss is the mean absolute error between each element in the input X and target Y. Now, your extracurricular measure might have guessed is to read through the documentation for the different loss functions, especially L1 loss. But for the sake of this video, let's just implement it for ourselves. Oh, and if you want a little bit of a graphic, I've got one here. This is where
5:53:22
we're up to, by the way, picking a loss function optimizer for step two. This is a fun part, right? We're getting into training a model. So we've got mean absolute error. Here's that graph we've seen before. Oh, look at this. Okay. So we've got the difference here. I've actually measured this before in the past. So I kind of knew what it was. Mean absolute error is if we repeat for all samples in our set that we're working with. And if we take the absolute difference between
5:53:50
these two dots, well, then we take the mean, we've got mean absolute error. So MAE loss equals torch mean we could write it out. That's the beauty of pine torch, right? We could write this out. Or we could use the torch and N version, which is recommended. So let's jump in. There's a colorful slide describing what we're about to do. So let's go set up a loss function. And then we're also going to put in here, set up an optimizer. So let's call it loss FN equals NN dot L1 loss.
5:54:29
Simple as that. And then if we have a look at what's our loss function, what does this say?
5:54:34
Oh my goodness. My internet is going quite slow today.
5:54:38
It's raining outside. So there might be some delays somewhere. But that's right. Gives us a chance to sit here and be mindful about what we're doing. Look at that. Okay. Loss function. L1 loss. Beautiful. So we've got a loss function. Our objective for training a machine learning model will be two. Let's go back. Look at the colorful graphic will be to minimize these distances here. And in turn, minimize the overall value of MAE. That is our goal.
5:55:05
If our red dots line up with our green dots, we will have a loss value of zero, the ideal point for a model to be. And so let's go here. We now need an optimizer. As we discussed before, the optimizer takes into account the loss of a model. So these two work in tandem. That's why I've put them as similar steps if we go back a few slides.
5:55:28
So this is why I put these as 2.1. Often picking a loss function and optimizer and pytorch come as part of the same package because they work together. The optimizer's objective is to give the model values. So parameters like a weight and a bias that minimize the loss function. They work in tandem. And so let's see what an optimizer optimizes. Where might that be? What if we search here? I typically don't use this search because I prefer just using Google
5:55:59
search. But does this give us optimizer? Hey, there we go. So again, pytorch has torch.optim
5:56:09
which is where the optimizers are. Torch.optim. Let me put this link in here.
5:56:17
This is another bit of your extracurricular. If you want to read more about different optimizers in pytorch, as you might have guessed, they have a few. Torch.optim is a package implementing various optimization algorithms. Most commonly used methods are already supported and the interface is general enough so that more sophisticated ones can also be easily integrated into the future. So if we have a look at what algorithms exist here, again, we're going to throw a lot of names
5:56:42
at you. But in the literature, a lot of them that have made it into here are already good working algorithms. So it's a matter of picking whichever one's best for your problem. How do you find that out? Well, SGD, stochastic gradient descent, is possibly the most popular. However, there are some iterations on SGD, such as Adam, which is another one that's really popular. So again, this is one of those other machine learning is part art, part science is trial and error of
5:57:16
figuring out what works best for your problem for us. We're going to start with SGD because it's the most popular. And if you were paying attention to a previous video, you might have seen that I said, look up gradient descent, wherever we got this gradient descent. There we go. So this is one of the main algorithms that improves our models. So gradient descent and back propagation. So if we have a look at this stochastic gradient descent, bit of a tongue twister,
5:57:43
is random gradient descent. So that's what stochastic means. So basically, our model improves by taking random numbers, let's go down here, here, and randomly adjusting them so that they minimize the loss. And once how optimizer, that's right here, once how optimizer torch dot opt in, let's implement SGD, SGD stochastic gradient descent. We're going to write this here, stochastic gradient descent. It starts by randomly adjusting these values. And once it's found
5:58:20
some random values or random steps that have minimized the loss value, we're going to see this in action later on, it's going to continue adjusting them in that direction. So say it says, oh, weights, if I increase the weights, it reduces the loss. So it's going to keep increasing the weights until the weights no longer reduce the loss. Maybe it gets to a point at say 0.65. If you increase the weights anymore, the loss is going to go up. So the optimizer is like,
5:58:48
well, I'm going to stop there. And then for the bias, the same thing happens. If it decreases the bias and finds that the loss increases, well, it's going to go, well, I'm going to try increasing the bias instead. So again, one last summary of what's going on here, a loss function measures how wrong our model is. And the optimizer adjust our model parameters, no matter whether there's two parameters or millions of them to reduce the loss. There are a couple of things that
5:59:15
an optimizer needs to take in. It needs to take in as an argument, params. So this is if we go to SGD, I'm just going to link this as well. SGD, there's the formula of what SGD does. I look at this and I go, hmm, there's a lot going on here. And take me a while to understand that. So I like to see it in code. So we need params. This is short for what parameters should I optimize as an optimizer. And then we also need an LR, which stands for, I'm going to write this in a comment, LR equals
5:59:49
learning rate, possibly the most, oh, I didn't even type rate, did I possibly the most important hyper parameter you can set? So let me just remind you, I'm throwing lots of words out here, but I'm kind of like trying to write notes about what we're doing. Again, we're going to see these in action in a second. So check out our models and parameters. So a parameter is a value that the model sets itself. So learning rate equals possibly the most important learning hyper parameter. I don't
6:00:32
need learning there, do I? Hyper parameter. And a hyper parameter is a value that us as a data scientist or a machine learning engineer set ourselves, you can set. So the learning rate is, in our case, let's go 0.01. You're like, Daniel, where did I get this value from? Well, again, these type of values come with experience. I think it actually says it in here, LR, LR 0.1. Yeah, okay, so the default is 0.1. But then if we go back to Optim, I think I saw it somewhere. Did I see it somewhere?
6:01:07
0.0? Yeah, there we go. Yeah, so a lot of the default settings are pretty good in torch optimizers. However, the learning rate, what does it actually do? We could go 0.01. These are all common values here. Triple zero one. I'm not sure exactly why. Oh, model, it's model zero. The learning rate says to our optimizer, yes, it's going to optimize our parameters here. But the higher the learning rate, the more it adjusts each of these parameters in one hit. So let's say it's 0.01. And it's going
6:01:43
to optimize this value here. So it's going to take that big of a step. If we changed it to here, it's going to take a big step on this three. And if we changed it to all the way to the end 0.01, it's only going to change this value. So the smaller the learning rate, the smaller the change in the parameter, the larger the learning rate, the larger the change in the parameter. So we've set up a loss function. We've set up an optimizer. Let's now move on to the next step
6:02:13
in our training workflow. And that's by building a training loop. Far out. This is exciting. I'll see you in the next video. Welcome back. In the last video, we set up a loss function. And we set up an optimizer. And we discussed the roles of each. So loss function measures how wrong our model is. The optimizer talks to the loss function and goes, well, if I change these parameters a certain way, does that reduce the loss function at all? And if it does, yes, let's keep adjusting them in
6:02:47
that direction. If it doesn't, let's adjust them in the opposite direction. And I just want to show you I added a little bit of text here just to concretely put down what we were discussing. Inside the optimizer, you'll often have to set two parameters, params and lr, where params is the model parameters you'd like to optimize for an example, in our case, params equals our model zero parameters, which were, of course, a weight and a bias. And the learning rate, which is lr
6:03:16
in optimizer, lr stands for learning rate. And the learning rate is a hyper parameter. Remember, a hyper parameter is a value that we the data scientist or machine learning engineer sets, whereas a parameter is what the model sets itself defines how big or smaller optimizer changes the model parameters. So a small learning rate, so the smaller this value results in small changes, a large learning rate results in large changes. So another question might be,
6:03:47
well, very valid question. Hey, I put this here already, is which loss function and which optimizer should I use? So this is another tough one, because it's problem specific. But with experience and machine learning, I'm showing you one example here, you'll get an idea of what works for your particular problem for a regression problem, like ours, a loss function of l1 loss, which is mai and pytorch. And an optimizer like torch dot opt in slash s gd like sarcastic gradient descent
6:04:14
will suffice. But for a classification problem, we're going to see this later on. Not this one specifically, whether a photo is a cat of a dog, that's just an example of a binary classification problem, you might want to use a binary classification loss. But with that being said, we now are moving on to, well, here's our whole goal is to reduce the MAE of our model. Let's get the workflow. We've done these two steps. Now we want to build a training loop. So
6:04:40
let's get back into here. There's going to be a fair few steps going on. We've already covered a few, but hey, nothing we can't handle together. So building a training loop in pytorch.
6:04:56
So I thought about just talking about what's going on in the training loop, but we can talk about the steps after we've coded them. How about we do that? So we want to build a training loop and a testing loop. How about we do that? So a couple of things we need in a training loop. So there's going to be a fair few steps here if you've never written a training loop before, but that is completely fine because you'll find that the first couple of times that you write this,
6:05:25
you'll be like, oh my gosh, there's too much going on here. But then when you have practice, you'll go, okay, I see what's going on here. And then eventually you'll write them with your eyes closed. I've got a fun song for you to help you out remembering things. It's called the unofficial pytorch optimization loop song. We'll see that later on, or actually, I'll probably leave that as an extension, but you'll see that you can also functionize these things, which we will do
6:05:48
later in the course so that you can just write them once and then forget about them. But we're going to write it all from scratch to begin with so we know what's happening. So we want to, or actually step zero, is loop through the data. So we want to look at the data multiple times because our model is going to, at first, start with random predictions on the data, make some predictions. We're trying to improve those. We're trying to minimize the loss to make those
6:06:15
predictions. We do a forward pass. So forward pass. Why is it called a forward pass? So this involves data moving through our model's forward functions. Now that I say functions because there might be plural, there might be more than one. And the forward method recall, we wrote in our model up here. Ford. A forward pass is our data going through this function here. And if you want to look at it visually, let's look up a neural network graphic. Images, a forward pass is just
6:06:55
data moving from the inputs to the output layer. So starting here input layer moving through the model. So that's a forward pass, also called forward propagation. Another time we'll have more than one name is used for the same thing. So we'll go back down here, forward pass. And I'll just write here also called forward propagation, propagation. Wonderful. And then we need to calculate the loss. So forward pass. Let me write this. To calculate or to make predictions,
6:07:33
make predictions on data. So calculate the loss, compare forward pass predictions. Oh, there's an undergoing in the background here of my place. We might be in for a storm. Perfect time to write code, compare forward pass predictions to ground truth labels. We're going to see all this in code in a second, calculate the loss. And then we're going to go optimise a zero grad. We haven't spoken about what this is, but that's okay. We're going to see that in a second. I'm not going to
6:08:04
put too much there. Loss backward. We haven't discussed this one either. There's probably three steps that we haven't really discussed. We've discussed the idea behind them, but not too much in depth. Optimise our step. So this one is loss backwards is move backwards. If the forward pass is forwards, like through the network, the forward pass is data goes into out. The backward pass data goes, our calculations happen backwards. So we'll see what that is in a second. Where were
6:08:35
we over here? We've got too much going on. I'm getting rid of these moves backwards through
6:08:41
the network to calculate the gradients. Oh, oh, the gradients of each of the parameters
6:08:53
of our model with respect to the loss. Oh my gosh, that is an absolute mouthful, but that'll do for now. Optimise a step. This is going to use the optimiser to adjust our model's parameters to try and improve the loss. So remember how I said in a previous video that I'd love you to watch the two videos I linked above. One on gradient descent and one on back propagation. If you did, you might have seen like there's a fair bit of math going on in
6:09:25
there. Well, that's essentially how our model goes from random parameters to better parameters, using math. Many people, one of the main things I get asked from machine learning is how do I learn machine learning if I didn't do math? Well, the beautiful thing about PyTorch is that it implements a lot of the math of back propagation. So this is back propagation. I'm going to write this down here. This is an algorithm called back, back propagation, hence the loss backward. We're
6:09:53
going to see this in code in a second, don't you worry? And this is gradient descent. So these two algorithms drive the majority of our learning. So back propagation, calculate the gradients of the parameters of our model with respect to the loss function and optimise our step, we'll trigger code to run gradient descent, which is to minimise the gradients because what is a gradient? Let's look this up. What is a gradient? I know we haven't written a code yet, but we're
6:10:24
going to do that. Images. Gradient, there we go. Changing y, changing x. Gradient is from high school math. Gradient is a slope. So if you were on a hill, let's find a picture of a hill. Picture of a hill. There we go. This is a great big hill. So if you were on the top of this hill, and you wanted to get to the bottom, how would you get to the bottom? Well, of course, you just walked down the hill. But if you're a machine learning model, what are you trying to do? Let's
6:11:00
imagine your loss is the height of this hill. You start off with your losses really high, and you want to take your loss down to zero, which is the bottom, right? Well, if you measure the gradient of the hill, the bottom of the hill is in the opposite direction to where the gradient is steep. Does that make sense? So the gradient here is an incline. We want our model to move towards the gradient being nothing, which is down here. And you could argue, yeah, the gradient's probably
6:11:27
nothing up the top here, but let's just for argument's sake say that we want to get to the bottom of the hill. So we're measuring the gradient, and one of the ways an optimisation algorithm works is it moves our model parameters so that the gradient equals zero, and then if the gradient of the loss equals zero, while the loss equals zero two. So now let's write some code. So we're going to set up a parameter called or a variable called epochs. And we're going to start with one,
6:11:53
even though this could be any value, let me define these as we go. So we're going to write code to do all of this. So epochs, an epoch is one loop through the data dot dot dot. So epochs, we're going to start with one. So one time through all of the data, we don't have much data. And so for epoch, let's go this, this is step zero, zero, loop through the data. By the way, when I say loop through the data, I want you to do all of these steps within the loop. And do dot dot dot
6:12:33
loop through the data. So for epoch in range epochs, even though it's only going to be one, we can adjust this later. And because epochs, we've set this ourselves, it is a,
6:12:45
this is a hyper parameter, because we've set it ourselves. And I know you could argue that, hey, our machine learning parameters of model zero, or our model parameters, model zero aren't actually parameters, because we've set them. But in the models that you build in the future, they will likely be set automatically rather than you setting them explicitly like we've done when we created model zero. And oh my gosh, this is taking quite a while to run. That's all right.
6:13:17
We don't need it to run fast. We just, we need to write some more code, then you'll come on. There's a step here I haven't discussed either. Set the model to training mode. So pytorch models have a couple of different modes. The default is training mode. So we can set it to training mode by going like this. Train. So what does train mode do in a pytorch model? My goodness. Is there a reason my engineer is going this slide? That's all right. I'm just going to
6:13:44
discuss this with talking again list. Train mode. Train mode in pytorch sets. Oh, there we go. Requires grad equals true. Now I wonder if we do with torch dot no grad member no grad is similar to inference mode. Will this adjust? See, I just wanted to take note of requires grad equals true. Actually, what I might do is we do this in a different cell. Watch this. This is just going to be rather than me just spit words at you. I reckon we might be able to get it work in doing
6:14:24
this. Oh, that didn't list the model parameters. Why did that not come out? Model zero dot eval.
6:14:33
So there's two modes of our mode and train mode model dot eval parameters. Hey, we're experimenting together on the fly here. And actually, this is what I want you to do is I want you to experiment with different things. It's not going to say requires grad equals false. Hmm. With torch dot no grad. Model zero dot parameters. I don't know if this will work, but it definitely works behind the scenes. And what I mean by works behind the scenes are not here. It works behind the scenes
6:15:07
when calculations have been made, but not if we're trying to explicitly print things out.
6:15:12
Well, that's an experiment that I thought was going to work and it didn't work. So train mode in pytorch sets all parameters that require gradients to require gradients.
6:15:27
So do you remember with the picture of the hill? I spoke about how we're trying to minimize the gradient. So the gradient is the steepness of the hill. If the height of the hill is a loss function and we want to take that down to zero, we want to take the gradient down to zero. So same thing with the gradients of our model parameters, which are here with respect to the loss function, we want to try and minimize that gradient. So that's gradient descent is take that gradient down to
6:15:54
zero. So model dot train. And then there's also model zero dot a vowel. So turns off gradient tracking. So we're going to see that later on. But for now, I feel like this video is getting far too long. Let's finish the training loop in the next video. I'll see you there.
6:16:19
Friends, welcome back. In the last video, I promised a lot of code, but we didn't get there. We discussed some important steps. I forgot how much behind the scenes there is to apply towards training loop. And I think it's important to spend the time that we did discussing what's going on, because there's a fair few steps. But once you know what's going on, I mean, later on, we don't have to write all the code that we're going to write in this video, you can functionize it. We're
6:16:42
going to see that later on in the course, and it's going to run behind the scenes for us. But we're spending a fair bit of time here, because this is literally the crux of how our model learns. So let's get into it. So now we're going to implement the forward pass, which involves our model's forward function, which we defined up here. When we built our model, the forward pass runs through this code here. So let's just write that. So in our case, because we're training, I'm just
6:17:12
going to write here. This is training. We're going to see dot of our later on. We'll talk about that when it comes. Let's do the forward pass. So the forward pass, we want to pass data through our model's forward method. We can do this quite simply by going y pred. So y predictions, because remember, we're trying to use our ideal model is using x test to predict y test on our test data set. We make predictions on our test data set. We learn on our training data set.
6:17:44
So we're passing, which is going to get rid of that because we don't need that. So we're passing our model x train and model zero is going to be our current model. There we go. So we learn patterns on the training data to evaluate our model on the test data. Number two, where we are.
6:18:05
So we have to calculate the loss. Now, in a previous video, we set up a loss function. So this is going to help us calculate the what what kind of loss are we using? We want to calculate the MAE. So the difference or the distance between our red dot and a green dot. And the formula would be the same if we had 10,000 red dots and 10,000 green dots, we're calculating how far they are apart. And then we're taking the mean of that value. So let's go back here. So calculate the loss.
6:18:38
And in our case, we're going to set loss equal to our loss function, which is L one loss in PyTorch, but it is MAE. Y-pred and Y-train. So we're calculating the difference between our models predictions on the training data set and the ideal training values. And if you want to go into torch dot NN loss functions, that's going to show you the order because sometimes this confuses me to what order the values go in here, but it goes prediction first, then labels and I may be wrong
6:19:14
there because I get confused here. My dyslexia kicks in, but I'm pretty sure it's predictions first, then actual labels. Do we have an example of where it's used? Yeah, import first, target next. So there we go. And truth be told, because it's mean absolute error, it shouldn't actually matter too much. But in the case of staying true to the documentation, let's do inputs first and then targets next for the rest of the course. Then we're going to go optimizer zero grad. Hmm,
6:19:48
haven't discussed this one, but that's okay. I'm going to write the code and then I'm going to discuss what it does. So what does this do? Actually, before we discuss this, I'm going to write these two steps because they kind of all work together. And it's a lot easier to discuss what optimizer zero grad does in the context of having everything else perform back propagation
6:20:14
on the loss with respect to the parameters of the model. Back propagation is going to take the loss value. So lost backward, I always say backwards, but it's just backward. That's the code there. And then number five is step the optimizer. So perform gradient descent. So optimizer dot step. Oh, look at us. We just wrote the five major steps of a training loop. Now let's discuss how all of these work together. So it's kind of strange, like the ordering of these, you might
6:20:54
think, Oh, what should I do the order? Typically the forward pass and the loss come straight up. Then there's a little bit of ambiguity around what order these have to come in. But the optimizer step should come after the back propagation. So I just like to keep this order how it is because this works. Let's just keep it that way. But what happens here? Well, it also is a little bit confusing in the first iteration of the loop because we've got zero grad. But what happens here is
6:21:23
that the optimizer makes some calculations in how it should adjust model parameters with regards to the back propagation of the loss. And so by default, these will by default, how the optimizer changes will accumulate through the loop. So we have to zero them above in step three for the next iteration of the loop. So a big long comment there. But what this is saying is, let's say we go through the loop and the optimizer chooses a value of one, change it by one. And
6:22:06
then it goes through a loop again, if we didn't zero it, if we didn't take it to zero, because that's what it is doing, it's going one to zero, it would go, okay, next one, two, three, four, five, six, seven, eight, all through the loop, right? Because we're looping here. If this was 10, it would accumulate the value that it's supposed to change 10 times. But we want it to start again, start fresh each iteration of the loop. And now the reason why it accumulates, that's
6:22:32
pretty deep in the pytorch documentation. From my understanding, there's something to do with like efficiency of computing. If you find out what the exact reason is, I'd love to know. So we have to zero it, then we perform back propagation. If you recall, back propagation is discussed in here. And then with optimizer step, we form gradient descent. So the beauty of pytorch, this is the beauty of pytorch, is that it will perform back propagation, we're going to have a
6:23:02
look at this in second, and gradient descent for us. So to prevent this video from getting too long, I know we've just written code, but I would like you to practice writing a training loop yourself, just write this code, and then run it and see what happens. Actually, you can comment this out, we're going to write the testing loop in a second. So your extra curriculum for this video is to, one, rewrite this training loop, is to, two, sing the pytorch optimization loop
6:23:31
song, let's go into here. If you want to remember the steps, well, I've got a song for you. This is the training loop song, we haven't discussed the test step, but maybe you could try this yourself. So this is an old version of the song, actually, I've got a new one for you. But let's sing this together. It's training time. So we do the forward pass, calculate the loss, optimise a zero grad, loss backwards, optimise a step, step, step. Now you only have to call optimise a step once,
6:24:01
this is just for jingle purposes. But for test time, let's test with torch no grad, do the forward pass, calculate the loss, watch it go down, down, down. That's from my Twitter, but this is a way that I help myself remember the steps that are going on in the code here. And if you want the video version of it, well, you're just going to have to search unofficial pytorch optimisation loop song. Oh, look at that, who's that guy? Well, he looks pretty cool. So I'll let you check that
6:24:38
out in your own time. But for now, go back through the training loop steps. I've got a colorful graphic coming up in the next video, we're going to write the testing steps. And then we're going to go back one more time and talk about what's happening in each of them. And again, if you'd like some even more extra curriculum, don't forget the videos I've shown you on back propagation and gradient descent. But for now, let's leave this video here. I'll see you in the next one.
6:25:05
Friends, welcome back. In the last few videos, we've been discussing the steps in a training loop in pytorch. And there's a fair bit going on. So in this video, we're going to step back through what we've done just to recap. And then we're going to get into testing. And it's nice and early where I am right now. The sun's about to come up. It's a very, very beautiful morning to be writing code. So let's jump in. We've got a little song here for what we're doing in the
6:25:34
training steps. For an epoch in a range, comodel.train, do the forward pass, calculate the loss of the measure zero grad, last backward of the measure step step step. That's the little jingle I use to remember the steps in here, because the first time you write it, there's a fair bit going on. But subsequent steps and subsequent times that you do write it, you'll start to memorize this. And even better later on, we're going to put it into a function so that we can just call it
6:26:06
over and over and over and over again. With that being said, let's jump in to a colorful slide, because that's a lot of code on the page. Let's add some color to it, understand what's happening. That way you can refer to this and go, Hmm, I see what's going on now. So for the loop, this is why it's called a training loop. We step through a number of epochs. One epoch is a single forward pass through the data. So pass the data through the model for a number of epochs. Epox is a
6:26:36
hyper parameter, which means you could set it to 100, you could set it to 1000, you could set it to one as we're going to see later on in this video. We skip this step with the colors, but we put the model in we call model.train. This is the default mode that the model is in. Essentially, it sets up a whole bunch of settings behind the scenes in our model parameters so that they can track the gradients and do a whole bunch of learning behind the scenes with these
6:27:07
functions down here. PyTorch does a lot of this for us. So the next step is the forward pass. We perform a forward pass on the training data in the training loop. This is an important note. In the training loop is where the model learns patterns on the training data. Whereas in the testing loop, we haven't got to that yet is where we evaluate the patterns that our model has learned or the parameters that our model has learned on unseen data. So we pass the data through the model,
6:27:36
this will perform the forward method located within the model object. So because we created a model object, you can actually call your models whatever you want, but it's good practice to you'll often see it just called model. And if you remember, we'll go back to the code. We created a forward method in our model up here, which is this, because our linear regression model, class, subclasses, nn.module, we need to create our own custom forward method. So that's why it's
6:28:06
called a forward pass is because not only does it, well, the technical term is forward propagation. So if we have a look at a neural network picture, forward propagation just means going through the network from the input to the output, there's a thing called back propagation, which we're going to discuss in a second, which happens when we call loss.backward, which is going backward through the model. But let's return to our colorful slide. We've done the forward pass, call a forward method,
6:28:40
which performs some calculation on the data we pass it. Next is we calculate the loss value, how wrong the model's predictions are. And this will depend on what loss function you use, what kind of predictions your model is outputting, and what kind of true values you have. But that's what we're doing here. We're comparing our model's predictions on the training data to what they should ideally be. And these will be the training labels. The next step, we zero
6:29:11
the optimizer gradients. So why do we do this? Well, it's a little confusing for the first epoch in the loop. But as we get down to optimizer dot step here, the gradients that the optimizer calculates accumulate over time so that for each epoch for each loop step, we want them to go back to zero. And now the exact reason behind why the optimizer accumulates gradients is buried somewhere within the pie torch documentation. I'm not sure of the exact reason from memory. It's because of
6:29:43
compute optimization. It just adds them up in case you wanted to know what they were. But if you find out exactly, I'd love to know. Next step is to perform back propagation on the loss function. That's what we're calling loss. backward. Now back propagation is we compute the gradient of every parameter with requires grad equals true. And if you recall, we go back to our code. We've set requires grad equals true for our parameters. Now the reason we've set requires
6:30:16
grad equals true is not only so back propagation can be performed on it. But let me show you what the gradients look like. So let's go loss function curve. That's a good idea. So we're looking for so we're looking for some sort of convex curve here. There we go. L two loss. We're using L one loss at the moment. Is there a better one here? All we need is just a nice looking curve. Here we go. So this is why we keep track of the gradients behind the scenes. Pie torch is going to create
6:30:51
some sort of curve for all of our parameters that looks like this. Now this is just a 2d plot. So the reason why we're just using an example from Google images is one, because you're going to spend a lot of your time Googling different things. And two, in practice, when you have your own custom neural networks, right now we only have two parameters. So it's quite easy to visualize a loss function curve like this. But when you have say 10 million parameters, you basically can't
6:31:21
visualize what's going on. And so pie torch again will take care of these things behind the scenes. But what it's doing is when we say requires grad pie torch is going to track the gradients of each of our parameters. And so what we're trying to do here with back propagation and subsequently gradient descent is calculate where the lowest point is. Because this is a loss function, this is MSC loss, we could trade this out to be MAE loss in our case or L1 loss for our specific
6:31:53
problem. But this is some sort of parameter. And we calculate the gradients because what is the gradient? Let's have a look. What is a gradient? A gradient is an inclined part of a road or railway. Now we want it in machine learning. What's it going to give us in machine learning, a gradient is a derivative of a function that has more than one input variable. Okay, let's dive in a little deeper. See, here's some beautiful loss landscapes. We're trying to get to the bottom of here. This
6:32:28
is what gradient descent is all about. So oh, there we go. So this is a cost function, which is also a loss function. We start with a random initial variable. What have we done? We started with a random initial variable. Right? Okay. And then we take a learning step. Beautiful. This is W. So this could be our weight parameter. Okay, we're connecting the dots here. This is exciting. We've got a lot of tabs here, but that's all right. We'll bring this all together in a second.
6:32:57
And what we're trying to do is come to the minimum. Now, why do we need to calculate the gradients? Well, the gradient is what? Oh, value of weight. Here we go. This is even better. I love Google images. So this is our loss. And this is a value of a weight. So we calculate the gradients. Why? Because the gradient is the slope of a line or the steepness. And so if we calculate the gradient here, and we find that it's really steep right up the top of this,
6:33:29
this incline, we might head in the opposite direction to that gradient. That's what gradient descent is. And so if we go down here, now, what are these step points? There's a little thing that I wrote down in the last video at the end of the last video I haven't told you about yet, but I was waiting for a moment like this. And if you recall, I said kind of all of these three steps optimizes zero grad loss backward, optimizes step are all together. So we calculate the
6:33:55
gradients because we want to head in the opposite direction of that gradient to get to a gradient value of zero. And if we get to a gradient value of zero with a loss function, well, then the loss is also zero. So that's why we keep track of a gradient with requires grad equals true. And again, PyTorch does a lot of this behind the scenes. And if you want to dig more into what's going on here, I'm going to show you some extra resources for back propagation,
6:34:20
which is calculating this gradient curve here, and gradient descent, which is finding the bottom of it towards the end of this video. And again, if we started over this side, we would just go in the opposite direction of this. So maybe this is a positive gradient here, and we just go in the opposite direction here. We want to get to the bottom. That is the main point of gradient descent.
6:34:42
And so if we come back, I said, just keep this step size in mind here. If we come back to where we created our loss function and optimizer, I put a little tidbit here for the optimizer. Because we've written a lot of code, and we haven't really discussed what's going on, but I like to do things on the fly as we need them. So inside our optimizer, we'll have main two parameters, which is params. So the model parameters you'd like to optimize,
6:35:11
params equals model zero dot parameters in our case. And then PyTorch is going to create something similar to this curve, not visually, but just mathematically behind the scenes for every parameter. Now, this is a value of weight. So this would just be potentially the weight parameter of our network. But again, if you have 10 million parameters, there's no way you could just create all of these curves yourself. That's the beauty of PyTorch. It's doing this behind the
6:35:36
scenes through a mechanism called torch autograd, which is auto gradient calculation. And there's beautiful documentation on this. If you'd like to read more on how it works, please go through that. But essentially behind the scenes, it's doing a lot of this for us for each parameter. That's the optimizer. Then within the optimizer, once we've told it what parameters to optimize, we have the learning rate. So the learning rate is another hyper parameter that defines how big or
6:36:04
small the optimizer changes the parameters with each step. So a small learning rate results in small changes, whereas a large learning rate is in large changes. And so if we look at this curve here, we might at the beginning start with large steps, so we can get closer and closer to the bottom. But then as we get closer and closer to the bottom, to prevent stepping over to this side of the curve, we might do smaller and smaller steps. And the optimizer in PyTorch,
6:36:34
there are optimizers that do that for us. But there is also another concept called learning rate scheduling, which is, again, something if you would like to look up and do more. But learning rate scheduling essentially says, hey, maybe start with some big steps. And then as we get closer and closer to the bottom, reduce how big the steps are that we take. Because if you've ever seen a coin, coin at the back of couch. This is my favorite analogy for this. If you've ever
6:37:04
tried to reach a coin at the back of a couch, like this excited young chap, if you're reaching towards the back of a couch, you take quite big steps as you say your arm was over here, you would take quite big steps until you get to about here. And in the closer you get to the coin, the smaller and smaller your steps are. Otherwise, what's going to happen? The coin is going to be lost. Or if you took two small steps, you'd never get to the coin. It would take forever to get there.
6:37:34
So that's the concept of learning rate. If you take two big steps, you're going to just end up over here. If you take two small steps, it's going to take you forever to get to the bottom here. And this bottom point is called convergence. That's another term you're going to come across. I know I'm throwing a lot of different terms at you, but that's the whole concept of the learning rate. How big is your step down here? In gradient descent. Gradient descent is this. Back propagation
6:37:59
is calculating these derivative curves or the gradient curves for each of the parameters in our model. So let's get out of here. We'll go back to our training steps. Where were we? I think we're up to back propagation. Have we done backward? Yes. So the back propagation is where we do the backward steps. So the forward pass, forward propagation, go from input to output. Back propagation, we take the gradients of the loss function with respect to each parameter in our model
6:38:30
by going backwards. That's what happens when we call loss.backward. PyTorch does that for us behind the scenes. And then finally, step number five is step the optimizer. We've kind of discussed that. As I said, if we take a step, let's get our loss curve back up. Loss function curve. Doesn't really matter what curve we use. The optimizer step is taking a step this way to try and optimize the parameters so that we can get down to the bottom here. And I also just noted
6:39:06
here that you can turn all of this into a function so we don't necessarily have to remember to write these every single time. The ordering of this, you'll want to do the forward pass first. And then calculate the loss because you can't calculate the loss unless you do the forward pass. I like this ordering here of these three as well. But you also want to do the optimizer step after the loss backward. So this is my favorite ordering. It works. If you like this ordering,
6:39:29
you can take that as well. With that being said, I think this video has gotten long enough. In the next video, I'd like to step through this training loop one epoch at a time so that we can see, I know I've just thrown a lot of words at you that this optimizer is going to try and optimize our parameters each step. But let's see that in action how our parameters of our model actually change every time we go through each one of these steps. So I'll see you in the next video.
6:40:00
Let's step through our model. Welcome back. And we've spent a fair bit of time on the training loop and the testing loop. Well, we haven't even got to that yet, but there's a reason behind this, because this is possibly one of the most important things aside from getting your data ready, which we're going to see later on in PyTorch deep learning is writing the training loop, because this is literally like how your model learns patterns and data. So that's why we're
6:40:27
spending a fair bit of time on here. And we'll get to the testing loop, because that's how you evaluate the patterns that your model has learned from data, which is just as important as learning the patterns themselves. And following on from the last couple of videos, I've just linked some YouTube videos that I would recommend for extra curriculum for back propagation, which is what happens when we call loss stop backward down here. And for the optimizer step,
6:40:54
gradient descent is what's happening there. So I've linked some extra resources for what's going on behind the scenes there from a mathematical point of view. Remember, this course focuses on writing PyTorch code. But if you'd like to dive into what math PyTorch is triggering behind the scenes, I'd highly recommend these two videos. And I've also added a note here as to which loss function and optimizer should I use, which is a very valid question. And again,
6:41:20
it's another one of those things that's going to be problem specific. But with experience over time, you work with machine learning problems, you write a lot of code, you get an idea of what works and what doesn't with your particular problem set. For example, like a regression problem, like ours, regression is again predicting a number. We use MAE loss, which PyTorch causes L1 loss. You could also use MSE loss and an optimizer like torch opt-in stochastic gradient
6:41:48
descent will suffice. But for classification, you might want to look into a binary classification, a binary cross entropy loss, but we'll look at a classification problem later on in the course. For now, I'd like to demonstrate what's going on in the steps here. So let's go model zero.
6:42:07
Let's look up the state dict and see what the parameters are for now. Now they aren't the original ones I don't think. Let's re-instantiate our model so we get re new parameters. Yeah, we recreated it here. I might just get rid of that. So we'll rerun our model code, rerun model state dict. And we will create an instance of our model and just make sure your parameters should be something similar to this. If it's not exactly like that, it doesn't
6:42:40
matter. But yeah, I'm just going to showcase you'll see on my screen what's going on anyway. State dict 3367 for the weight and 012888 for the bias. And again, I can't stress enough. We've only got two parameters for our model and we've set them ourselves future models that you build and later ones in the course will have much, much more. And we won't actually explicitly set any of them ourselves. We'll check out some predictions. They're going to be terrible because we're using
6:43:11
random parameters to begin with. But we'll set up a new loss function and an optimizer. Optimizer is going to optimize our model zero parameters, the weight and bias. The learning rate is 0.01, which is relatively large step. That would be a bit smaller. Remember, the larger the learning rate, the bigger the step, the more the optimizer will try to change these parameters every step. But let's stop talking about it. Let's see it in action. I've set a manual seed here too, by the way,
6:43:40
because the optimizer steps are going to be quite random as well, depending on how the models predictions go. But this is just to try and make it as reproduces possible. So keep this in mind, if you get different values to what we're going to output here from my screen to your screen, don't worry too much. What's more important is the direction they're going. So ideally, we're moving these values here. This is from we did one epoch before. We're moving these values
6:44:08
closer to the true values. And in practice, you won't necessarily know what the true values are. But that's where evaluation of your model comes in. We're going to cover that when we write a testing loop. So let's run one epoch. Now I'm going to keep that down there. Watch what happens. We've done one epoch, just a single epoch. We've done the forward pass. We've calculated the loss. We've done optimizer zero grad. We've performed back propagation. And we've stepped the optimizer.
6:44:39
What is stepping the optimizer do? It updates our model parameters to try and get them further closer towards the weight and bias. If it does that, the loss will be closer to zero. That's what it's trying to do. How about we print out the loss at the same time. Print loss and the loss.
6:45:00
Let's take another step. So the loss is 0301. Now we check the weights and the bias. We've changed again three, three, four, four, five, one, four, eight, eight. We go again. The loss is going down. Check it. Hey, look at that. The values are getting closer to where they should be if over so slightly.
6:45:25
Loss went down again. Oh my goodness, this is so amazing. Look, we're training our, let's print this out in the same cell. Print our model. State dict. We're training our first machine learning model here, people. This is very exciting, even if it's only step by step and it's only a small model. This is very important. Loss is going down again. Values are getting closer to where they should be. Again, we won't really know where they should be in real problems, but for
6:45:55
now we do. So let's just get excited. The real way to sort of measure your model's progress and practice is a lower loss value. Remember, lower is better. A loss value measures how wrong your model is. We're going down. We're going in the right direction. So that's what I meant by, as long as your values are going in the similar direction. So down, we're writing similar code here, but if your values are slightly different in terms of the exact numbers, don't worry too
6:46:21
much because that's inherent to the randomness of machine learning, because the steps that the optimizer are taking are inherently random, but they're sort of pushed in a direction. So we're doing gradient descent here. This is beautiful. How low can we get the loss? How about we try to get to 0.1? Look at that. We're getting close to 0.1. And then, I mean, we don't have to do this hand by hand. The bias is getting close to where it exactly should be. We're below 0.1.
6:46:51
Beautiful. So that was only about, say, 10 passes through the data, but now you're seeing it in practice. You're seeing it happen. You're seeing gradient descent. Let's go gradient descent work in action. We've got images. This is what's happening. We've got our cost function. J is another term for cost function, which is also our loss function. We start with an initial weight. What have we done? We started with an initial weight, this value here. And what are we doing?
6:47:20
We've measured the gradient pytorch has done that behind the scenes for us. Thank you pytorch. And we're taking steps towards the minimum. That's what we're trying to do. If we minimize the gradient of our weight, we minimize the cost function, which is also a loss function. We could keep going here for hours and get as long as we want. But my challenge for you, or actually, how about we make some predictions with our model we've got right now? Let's make some predictions.
6:47:47
So with torch dot inference mode, we'll make some predictions together. And then I'm going to set you a challenge. How about you run this code here for 100 epochs after this video, and then you make some predictions and see how that goes. So why preds? Remember how poor our predictions are? Why preds new equals, we just do the forward pass here. Model zero on the test data. Let's just remind ourselves quickly of how poor our previous predictions were.
6:48:16
Plot predictions, predictions equals y. Do we still have this saved? Why preds? Hopefully, this is still saved. There we go. Shocking predictions, but we've just done 10 or so epochs. So 10 or so training steps have our predictions. Do they look any better? Let's run this. We'll copy this code. You know my rule. I don't really like to copy code, but in this case, I just want to exemplify a point. I like to write all the code myself. What do we got? Why preds
6:48:47
new? Look at that. We are moving our predictions close at the red dots closer to the green dots. This is what's happening. We're reducing the loss. In other words, we're reducing the difference between our models predictions and our ideal outcomes through the power of back propagation and gradient descent. So this is super exciting. We're training our first machine learning model. My challenge to you is to run this code here. Change epochs to 100. See how low you can get this
6:49:17
loss value and run some predictions, plot them. And I think it's time to start testing. So give that a go yourself, and then we'll write some testing code in the next video. I'll see you there.
6:49:32
Welcome back. In the last video, we did something super excited. We saw our loss go down. So the loss is remember how different our models predictions are to what we'd ideally like them. And we saw our model update its parameters through the power of back propagation and gradient descent, all taken care of behind the scenes for us by PyTorch. So thank you, PyTorch. And again, if you'd like some extra resources on what's actually happening from a math perspective for back propagation and
6:50:03
gradient descent, I would refer to you to these. Otherwise, this is also how I learn about things. Gradient descent. There we go. How does gradient descent work? And then we've got back propagation.
6:50:15
And just to reiterate, I am doing this and just Googling these things because that's what you're going to do in practice. You're going to come across a lot of different things that aren't covered in this course. And this is seriously what I do day to day as a machine learning engineer if I don't know what's going on. Just go to Google, read, watch a video, write some code, and then I build my own intuition for it. But with that being said, I also issued you the challenge
6:50:43
of trying to run this training code for 100 epochs. Did you give that a go? I hope you did. And how low did your loss value? Did the weights and bias get anywhere close to where they should have been? How do the predictions look? Now, I'm going to save that for later on, running this code for 100 epochs. For now, let's write some testing code. And just a note, you don't necessarily have to write the training and testing loop together. You can functionize them, which we will be doing later
6:51:11
on. But for the sake of this intuition, building and code practicing and first time where we're writing this code together, I'm going to write them together. So testing code, we call model.ofour, what does this do? So this turns off different settings in the model not needed for evaluation slash testing. This can be a little confusing to remember when you're writing testing code. But we're going to do it a few times until it's habit. So just make it a habit. If you're training your
6:51:44
model, call model dot train to make sure it's in training mode. If you're testing or evaluating your model. So that's what a vowel stands for evaluate, call model dot a vowel. So it turns off different settings in the model not needed for evaluation. So testing, this is things like drop out. We haven't seen what drop out is slash batch norm layers. But if we go into torch dot and end, I'm sure you'll come across these things in your future machine learning endeavors. So drop
6:52:12
out drop out layers. There we go. And batch norm. Do we have batch batch norm? There we go. If you'd like to work out what they are, feel free to check out the documentation. Just take it from me for now that model of our turns off different settings not needed for evaluation and testing. Then we set up with torch dot inference mode, inference mode. So what does this do? Let's write down here.
6:52:42
So this turns off gradient tracking. So as we discussed, if we have parameters in our model, and it turns off actually a few more things and a couple more things behind the scenes,
6:52:57
these are things again, not needed for testing. So we discussed that if parameters in our model have requires grad equals true, which is the default for many different parameters in pytorch, pytorch will behind the scenes keep track of the gradients of our model and use them in lost up backward and optimizer step for back propagation and gradient descent. However,
6:53:21
we only need those two back propagation and gradient descent during training because that is when our model is learning. When we are testing, we are just evaluating the parameters the patterns that our model has learned on the training data set. So we don't need to do any learning when we're testing. So we turn off the things that we don't need. And is this going to have the correct spacing for me? I'm not sure we'll find out. So we still do the forward pass
6:53:49
in testing mode, do the forward pass. And if you want to look up torch inference mode, just go torch inference mode. There's a great tweet about it that pytorch did, which explains what's happening. I think we've covered this before, but yeah, want to make your inference code and pytorch run faster. Here's a quick thread on doing exactly that. So inference mode is torch no grad. Again, you might see torch no grad. I think I'll write that down just to
6:54:17
let you know. But here's what's happening behind the scenes. A lot of optimization code, which is beautiful. This is why we're using pytorch so that our code runs nice and far. Let me go there. You may also see with torch dot no grad in older pytorch code. It does similar things, but inference mode is the faster way of doing things according to the thread. And according to there's a blog post attached to there as well, I believe.
6:54:44
So you may also see torch dot no grad in older pytorch code, which would be valid. But again, inference mode is the better way of doing things. So do forward pass. So let's get our model. We want to create test predictions here. So we're going to go model zero. There's a lot of code going on here, but I'm going to just step by step it in a second. We'll go back through it all. And then number two is calculate the loss. Now we're doing the test predictions here,
6:55:17
calculate the loss test predictions with model zero. So now we want to calculate the what we want to calculate the test loss. So this will be our loss function, the difference between the test pred and the test labels. That's important. So for testing, we're working with test data, for training, we're working with training data. Model learns patterns on the training data, and it evaluates those patterns that it's learned, the different parameters on the testing data. It
6:55:48
has never seen before, just like in a university course, you'd study the course materials, which is the training data, and you'd evaluate your knowledge on materials you'd hopefully never seen before, unless you sort of were friends with your professor, and they gave you the exam before the actual exam that would be cheating right. So that's a very important point for the test data set. Don't let your model see the test data set before you evaluate it. Otherwise, you'll get
6:56:15
poor results. And that's putting it out what's happening. Epoch, we're going to go Epoch, and then I will introduce you to my little jingle to remember all of these steps because there's a lot going on. Don't you worry. I know there's a lot going on, but again, with practice, we're going to know what's happening here. Like it's the back of our hand. All right. So do we need this? Oh, yeah, we could say that. Oh, no, we don't need test here. Loss. This is
6:56:49
loss, not test. Print out what's happening. Okay. And we don't actually need to do this every epoch. We could just go say if epoch divided by 10 equals zero, print out what's happening. Let's do that rather than clutter everything up, print it out, and we'll print out this.
6:57:12
So let's just step through what's happening. We've got 100 epochs. That's what we're about to run, 100 epochs. Our model is trained for about 10 so far. So it's got a good base. Maybe we'll just get rid of that base. Start a new instance of our model. So we'll come right back down.
6:57:33
So our model is back to randomly initialized parameters, but of course, randomly initialized flavored with a random seed of 42. Lovely, lovely. And so we've got our training code here. We've discussed what's happening there. Now, we've got our testing code. We call model dot eval, which turns off different settings in the model, not needed for evaluation slash testing. We call with torch inference mode context manager, which turns off gradient tracking and a couple more
6:57:59
things behind the scenes to make our code faster. We do the forward pass. We do the test predictions. We pass our model, the test data, the test features to calculate the test predictions. Then we calculate the loss using our loss function. We can use the same loss function that we used for the training data. And it's called the test loss, because it's on the test data set. And then we print out what's happening, because we want to know what's happening while our
6:58:25
model's training, we don't necessarily have to do this. But the beauty of PyTorch is you can use basic Python printing statements to see what's happening with your model. And so, because we're doing 100 epochs, we don't want to clutter up everything here. So we'll just print out what's happening every 10th epoch. Again, you can customize this as much as you like what's printing out here. This is just one example. If you had other metrics here, such as calculating
6:58:49
model accuracy, we might see that later on, hint hint. We might print out our model accuracy. So this is very exciting. Are you ready to run 100 epochs? How low do you think our loss can go?
6:59:02
This loss was after about 10. So let's just save this here. Let's give it a go. Ready? Three, two, one. Let's run. Oh my goodness. Look at that. Waits. Here we go. Every 10 epochs were printing out what's happening. So the zero epoch, we started with losses 312. Look at it go down. Yes, that's what we want. And our weights and bias, are they moving towards our ideal weight and bias values of 0.7 and 0.3? Yes, they're moving in the right direction here. The loss is
6:59:34
going down. Epoch 20, wonderful. Epoch 30, even better. 40, 50, going down, down, down. Yes, this is what we want. This is what we want. Now, we're predicting a straight line here. Look how low the loss gets. After 100 epochs, we've got about three times less than what we had before.
6:59:55
And then we've got these values are quite close to where they should be, 0.5629, 0.3573. We'll make some predictions. What do they look like? Why preds new? This is the original predictions with random values. And if we make why preds new, look how close it is after 100 epochs. Now, what's our, do we print out the test loss? Oh no, we're printing out loss as well. Let's get rid of that. I think this is this. Yeah, that's this statement here. Our code would have
7:00:25
been a much cleaner if we didn't have that, but that's all right. Life goes on. So our test loss, because this is the test predictions that we're making, is not as low as our training loss.
7:00:36
I wonder how we could get that lower. What do you think we could do? We just trained it for longer. And what happened? How do you think you could get these red dots to line up with these green dots? Do you think you could? So that's my challenge to you for the next video. Think of something that you could do to get these red dots to match up with these green dots, maybe train for longer. How do you think you could do that? So give that a shot. And I'll see
7:01:01
in the next video, we'll review what our testing code is doing. I'll see you there.
7:01:10
Welcome back. In the last video, we did something super exciting. We trained our model for 100 epochs and look how good the predictions got. But I finished it off with challenging you to see if you could align the red dots with the green dots. And it's okay if you're not sure how the best way to do that. That's what we're here for. We're here to learn what are the best way to do these things together. But you might have had the idea of potentially training the model for a little bit
7:01:36
longer. So how could we do that? Well, we could just rerun this code. So the model is going to remember the parameters that it has from what we've done here. And if we rerun it, well, it's going to start from where it finished off, which is already pretty good for our data set. And then it's going to try and improve them even more. This is, I can't stress enough, like what we are doing here is going to be very similar throughout the entire rest of the course for training more
7:02:04
and more models. So this step that we've done here for training our model and evaluating it is seriously like the fundamental steps of deep learning with PyTorch is training and evaluating a model. And we've just done it. Although I'll be it to predict some red dots and green dots.
7:02:25
That's all right. So let's try to line them up, hey, red dots onto green dots. I reckon if we train it for another 100 epochs, we should get pretty darn close. Ready? Three, two, one. I'm going to run this cell again. Runs really quick because our data's nice and simple. But look at this, lastly, we started 0244. Where do we get down to? 008. Oh my goodness. So we've improved it by another three X or so. And now this is where our model has got really good.
7:02:55
On the test loss, we've gone from 00564. We've gone down to 005. So almost 10X improvement there. And so we make some more predictions. What are our model parameters? Remember the ideal ones here. We won't necessarily know them in practice, but because we're working with a simple data set, we know what the ideal parameters are. Model zero state dig weights. These are what they previously were. What are they going to change to? Oh, would you look at that? Oh,
7:03:27
06990. Now, again, if yours are very slightly different to mine, don't worry too much. That is the inherent randomness of machine learning and deep learning. Even though we set a manual seed, it may be slightly different. The direction is more important. So if your number here is not exactly what mine is, it should still be quite close to 0.7. And the same thing with this one. If it's not exactly what mine is, don't worry too much. The same with all of these loss values
7:03:57
as well. The direction is more important. So we're pretty darn close. How do these predictions look? Remember, these are the original ones. We started with random. And now we've trained a model. So close. So close to being exactly that. So a little bit off. But that's all right. We could tweak a few things to improve this. But I think that's well and truly enough for this example purpose. You see what's happened. Of course, we could just create a model and set the parameters
7:04:26
ourselves manually. But where would be the fun in that? We just wrote some machine learning code to do it for us with the power of back propagation and gradient descent. Now in the last video, we wrote the testing loop. We discussed a few other steps here. But now let's go over it with a colorful slide. Hey, because I mean, code on a page is nice, but colors are even nicer. Oh, we haven't done this. We might set up this in this video too. But let's just discuss what's going on.
7:04:56
Create an empty list for storing useful value. So this is helpful for tracking model progress. How can we just do this right now? Hey, we'll go here and we'll go.
7:05:07
So what did we have? Epoch count equals that. And then we'll go
7:05:14
lost values. So why do we keep track of these? It's because
7:05:21
if we want to monitor our models progress, this is called tracking experiments. So track different values. If we wanted to try and improve upon our current model with a future model. So our current results, such as this, if we wanted to try and improve upon it, we might build an entire other model. And we might train it in a different setup. We might use a different learning rate. We might use a whole bunch of different settings, but we track the values so that we
7:05:48
can compare future experiments to past experiments, like the brilliant scientists that we are. And so where could we use these lists? Well, we're calculating the loss here. And we're calculating the test loss here. So maybe we each time append what's going on here as we do a status update. So epoch count dot append, and we're going to go a current epoch. And then we'll go loss values dot append, a current loss value. And then we'll do test loss values dot append, the current test
7:06:34
loss values. Wonderful. And now let's re-instantiate our model so that it starts from fresh. So this is just create another instance. So we're just going to re-initialize our model parameters to start from zero. If we wanted to, we could functionize all of this so we don't have to go right back up to the top of the code. But just for demo purposes, we're doing it how we're doing it. And I'm going to run this for let's say 200 epochs, because that's what we ended up doing,
7:07:00
right? We ran it for 200 epochs, because we did 100 epochs twice. And I want to show you something beautiful, one of the most beautiful sites in machine learning. So there we go, we run it for 200 epochs, we start with a fairly high training loss value and a fairly high test loss value. So remember, what is our loss value? It's ma e. So if we go back, yeah, this is what we're measuring for loss. So this means for the test loss on average, each of our dot points here, the red
7:07:30
predictions are 0.481. That's the average distance between each dot point. And then ideally, what are we doing? We're trying to minimize this distance. That's the ma e. So the mean absolute error. And we get it right down to 0.05. And if we make predictions, what do we have here, we get very close to the ideal weight and bias, make our predictions, have a look at the new predictions. Yeah, very small distance here. Beautiful. That's a low loss value.
7:08:03
Ideally, they'd line up, but we've got as close as we can for now. So this is one of the most beautiful sites in machine learning. So plot the loss curves. So let's make a plot, because what we're doing, we were tracking the value of epoch count, loss values and test loss values. Let's have a look at what these all look like. So epoch count goes up, loss values ideally go down. So we'll get rid of that. We're going to create a plot p l t dot plot. We're going to step back
7:08:36
through the test loop in a second with some colorful slides, label equals train loss.
7:08:48
And then we're going to go plot. You might be able to tell what's going on here. Test loss values. We're going to visualize it, because that's the data explorer's motto, right, is visualize, visualize, visualize. This is equals. See, collab does this auto correct. That doesn't really work very well. And I don't know when it does it and why it doesn't. And we got, I know, we didn't, we didn't say loss value. So that's a good auto correct. Thank you, collab.
7:09:18
So training and loss and test loss curves. So this is another term you're going to come across often is a loss curve. Now you might be able to think about a loss curve. If we're doing a loss curve, and it's starting at the start of training, what do we want that curve to do?
7:09:36
What do we want our loss value to do? We want it to go down. So what should an ideal loss curve look like? Well, we're about to see a couple. Let's have a look. Oh, what do we got wrong? Well, we need to, I'll turn it into NumPy. Is this what we're getting wrong? So why is this wrong?
7:09:58
Loss values. Why are we getting an issue? Test loss values.
7:10:05
Ah, it's because they're all tens of values. So I think we should, let's, I might change this to NumPy. Oh, can I just do that? If I just call this as a NumPy array, we're going to try and fix this on the fly. People, NumPy array, we'll just turn this into a NumPy array. Let's see if we get NumPy. I'm figuring these things out together. NumPy as NumPy, because mapplotlib works with NumPy. Yeah, there we go. So can we do loss values? Maybe
7:10:45
I'm going to try one thing, torch dot tensor, loss values, and then call
7:10:53
CPU dot NumPy. See what happens here. There we go. Okay, so let's just copy this. So what we're doing here is our loss values are still on PyTorch, and they can't be because mapplotlib works with
7:11:14
NumPy. And so what we're doing here is we're converting our loss values of the training loss to NumPy. And if you call from the fundamental section, we call CPU and NumPy, I wonder if we can just do straight up NumPy, because we're not working on there. Yeah, okay, we don't need CPU because we're not working on the GPU yet, but we might need that later on. Well, this work. Beautiful. There we go. One of the most beautiful sides in machine learning is a declining loss
7:11:40
curve. So this is how we keep track of our experiments, or one way, quite rudimentary. We'd like to automate this later on. But I'm just showing you one way to keep track of what's happening. So the training loss curve is going down here. The training loss starts at 0.3, and then it goes right down. The beautiful thing is they match up. If there was a two bigger distance behind the train loss and the test loss, or sorry, between, then we're running into some problems. But if they
7:12:09
match up closely at some point, that means our model is converging and the loss is getting as close to zero as it possibly can. If we trained for longer, maybe the loss will go almost basically to zero. But that's an experiment I'll leave you to try to train that model for longer. Let's just step back through our testing loop to finish off this video. So we did that. We created empty lists for strong useful values, storing useful values, strong useful values. Told the
7:12:37
model what we want to evaluate or that we want to evaluate. So we put it in an evaluation mode. It turns off functionality used for training, but not evaluations, such as drop out and batch normalization layers. If you want to learn more about them, you can look them up in the documentation. Turn on torch inference mode. So this is for faster performance. So we don't necessarily need this, but it's good practice. So I'm going to say that yes, turn on torch inference mode. So this
7:13:03
disables functionality such as gradient tracking for inference. Gradient tracking is not needed for inference only for training. Now we pass the test data through the model. So this will call the models implemented forward method. The forward pass is the exact same as what we did in the training loop, except we're doing it on the test data. So big notion there, training loop, training data, testing loop, testing data. Then we calculate the test loss value,
7:13:32
how wrong the models predictions are on the test data set. And of course, lower is better. And finally, we print out what's happening. So we can keep track of what's going on during training. We don't necessarily have to do this. You can customize this print value to print out almost whatever you want, because it's pie torches, basically very beautifully interactive with pure Python. And then we keep track of the values of what's going on on epochs and train loss and test
7:14:00
loss. We could keep track of other values here. But for now, we're just going, okay, what's the loss value at a particular epoch for the training set? And for the test set. And of course, all of this could be put into a function. And that way we won't have to remember these steps off by heart. But the reason why we've spent so much time on this is because we're going to be using this training and test functionality for all of the models that we build throughout this course.
7:14:26
So give yourself a pat in the back for getting through all of these videos. We've written a lot of code. We've discussed a lot of steps. But if you'd like a song to remember what's happening, let's finish this video off with my unofficial PyTorch optimization loop song. So for an epoch in a range, go model dot train, do the forward pass, calculate the loss, optimize a zero grad, loss backward, optimize a step, step, step. No, you only have to call this once.
7:14:59
But now let's test, go model dot eval with torch inference mode, do the forward pass, calculate the loss. And then the real song goes for another epoch because you keep going back through. But we finish off with print out what's happening. And then of course, we evaluate what's going on. With that being said, it's time to move on to another thing. But if you'd like to review what's happening, please, please, please try to run this code for yourself again and check out the
7:15:29
slides and also check out the extra curriculum. Oh, by the way, if you want to link to all of the extra curriculum, just go to the book version of the course. And it's all going to be in here. So that's there ready to go. Everything I link is extra curriculum will be in the extra curriculum of each chapter. I'll see you in the next video. Welcome back. In the last video, we saw how to train our model and evaluate it by not only looking at the loss metrics and the loss curves,
7:16:03
but we also plotted our predictions and we compared them. Hey, have a go at these random predictions. Quite terrible. But then we trained a model using the power of back propagation and gradient descent. And now look at our predictions. They're almost exactly where we want them to be. And so you might be thinking, well, we've trained this model and it took us a while to write all this code to get some good predictions. How might we run that model again? So I've took
7:16:28
in a little break after the last video, but now I've come back and you might notice that my Google Colab notebook has disconnected. So what does this mean if I was to run this? Is it going to work? I'm going to connect to a new Google Colab instance. But will we have all of the code that we've run above? You might have already experienced this if you took a break before and came back to the videos. Ah, so plot predictions is no longer defined. And do you know what that means? That
7:16:59
means that our model is also no longer defined. So we would have lost our model. We would have lost all of that effort of training. Now, luckily, we didn't train the model for too long. So we can just go run time, run all. And it's going to rerun all of the previous cells and be quite quick.
7:17:17
Because we're working with a small data set and using a small model. But we've been through all of this code. Oh, what have we got wrong here? Model zero state dict. Well, that's all right. This is good. We're finding errors. So if you want to as well, you can just go run after. It's going to run all of the cells after. Beautiful. And we come back down. There's our model training. We're getting very similar values to what we got before. There's the lost curves. Beautiful.
7:17:42
Still going. Okay. Now our predictions are back because we've rerun all the cells and we've got our model here. So what we might cover in this video is saving a model in PyTorch. Because if we're training a model and you get to a certain point, especially when you have a larger model, you probably want to save it and then reuse it in this particular notebook itself. Or you might want to save it somewhere and send it to your friend so that your friend can try it out. Or you
7:18:13
might want to use it in a week's time. And if Google Colab is disconnected, you might want to be able to load it back in somehow. So now let's see how we can save our models in PyTorch. So
7:18:25
I'm going to write down here. There are three main methods you should know about
7:18:34
for saving and loading models in PyTorch because of course with saving comes loading. So we're going to over the next two videos discuss saving and loading. So one is torch.save. And as you might guess, this allows you to save a PyTorch object in Python's pickle format. So you may or may not be aware of Python pickle. There we go. Python object serialization. There we go. So we've got the pickle module implements a binary protocols or implements binary protocols for serializing
7:19:15
and deserializing a Python object. So serializing means I understand it is saving and deserializing means that it's loading. So this is what PyTorch uses behind the scenes, which is from pure Python. So if we go back here in Python's pickle format, number two is torch.load, which you might be able to guess what that does as well, allows you to load a saved PyTorch object. And number three is also very important is torch.nn.module.loadStatedict. Now what does this allow you to do? Well,
7:19:54
this allows you to load a model's saved dictionary or save state dictionary. Yeah, that's what we'll call it. Save state dictionary. Beautiful. And what's the model state dict? Well, let's have a look, model zero dot state dict. The beauty of PyTorch is that it stores a lot of your model's important parameters in just a simple Python dictionary. Now it might not be that simple because our model, again, only has two parameters. In the future, you may be working with models with millions of
7:20:25
parameters. So looking directly at the state deck may not be as simple as what we've got here. But the principle is still the same. It's still a dictionary that holds the state of your model. And so I've got these three methods I want to show you where from because this is going to be your extra curriculum, save and load models, your extra curriculum for this video.
7:20:50
If we go into here, this is a very, very, very important piece of PyTorch documentation, or maybe even a tutorial. So your extra curriculum for this video is to go through it. Here we go. We've got torch, save, torch, load, torch, module, state deck. That's where, or load state deck, that's where I've got the three things that we've just written down. And there's a fair few different pieces of information. So what is a state deck? So in PyTorch, the learnable
7:21:17
parameters, i.e. the weights and biases of a torch and end module, which is our model. Remember, our model subclasses and end module are contained in the model's parameters. Access with model.parameters, a state deck is simply a Python dictionary object that maps each layer to its parameter tensor. That's what we've seen. And so then if we define a model, we can initialize the model. And if we wanted to print the state decked, we can use that.
7:21:44
The optimizer also has a state deck. So that's something to be aware of. You can go optimizer.state deck. And then you get an output here. And this is our saving and loading model for inference. So inference, again, is making a prediction. That's probably what we want to do in the future at some point. For now, we've made predictions right within our notebook. But if we wanted to use our model outside of our notebook, say in an application, or in another notebook that's not this one,
7:22:11
you'll want to know how to save and load it. So the recommended way of saving and loading a PyTorch model is by saving its state deck. Now, there is another method down here, which is saving and loading the entire model. So your extracurricular for this lesson,
7:22:29
we're going to go through the code to do this. But your extracurricular is to read all of the
7:22:34
sections in here, and then figure out what the pros and cons are of saving and loading the entire model versus saving and loading just the state deck. So that's a challenge for you for this video. I'm going to link this in here. And now let's write some code to save our model. So PyTorch save and load code. Code tutorial plus extracurricular. So if we go
7:23:10
saving our PyTorch model. So what might we want? What do you think the save parameter takes? If we have torch.save, what do you think it takes inside it? Well, let's find out together. Hey, so let's import part lib. We're going to see why in a second. This is Python's module for dealing with writing file paths. So if we wanted to save something to this is Google Colab's file section over here. But just remember, if we do save this from within Google Colab,
7:23:41
the model will disappear if our Google Colab notebook instance disconnects. So I'll show you how to download it from Google Colab if you want. Google Colab also has a way save from Google Colab
7:23:57
Google Colab to Google Drive to save it to your Google Drive if you wanted to. But I'll leave you to look at that on your own if you like. So we're first going to create a model directory. So create models directory. So this is going to help us create a folder over here called models. And of course, we could create this by hand by adding a new folder here somewhere. But I like to do it with code. So model path, we're going to set this to path, which is using the path library
7:24:28
here to create us a path called models. Simple. We're just going to save all of our models to models to the models file. And then we're going to create model path, we're going to make that directory model path dot mkdir for make directory. We're going to set parents to equals true.
7:24:49
And we're also going to set exist okay equals to true. That means if it already existed, it won't throw us an error. It will try to create it. But if it already exists, it'll just recreate the parents directory or it'll leave it there. It won't error out on us. We're also going to create a model save path. This way, we can give our model a name. Right now, it's just model zero.
7:25:12
We want to save it under some name to the models directory. So let's create the model name.
7:25:18
Model name equals 01. I'm going to call it 01 for the section. That way, if we have more models later on the course, we know which ones come from where you might create your own naming convention, model workflow, pytorch workflow, model zero dot pth. And now this is another important point. Pytorch objects usually have the extension dot pth for pytorch or dot pth. So if we go in here, and if we look up dot pth, yeah, a common convention is to save models
7:25:52
using either a dot pth or dot pth file extension. I'll let you choose which one you like. I like dot pth. So if we go down here dot pth, they both result in the same thing. You just have to remember to make sure you write the right loading path and right saving path. So now we're going to create our model save path, which is going to be our model path. And because we're using the path lib, we can use this syntax that we've got here, model path slash model name. And then if we just print out
7:26:23
model save path, what does this look like? There we go. So it creates a supposic path using the path lib library of models slash 01 pytorch workflow model zero dot pth. We haven't saved our model there yet. It's just got the path that we want to save our model ready. So if we refresh this, we've got models over here. Do we have anything in there? No, we don't yet. So now is our step to save the model. So three is save the model state dict. Why are we saving the state
7:26:59
dict? Because that's the recommended way of doing things. If we come up here, saving and loading the model for inference, save and load the state dict, which is recommended. We could also save the entire model. But that's part of your extra curriculum to look into that. So let's use some syntax. It's quite like this torch dot save. And then we pass it an object. And we pass it a path of where to save it. We already have a path. And good thing is we already have a model. So we just have to call
7:27:24
this. Let's try it out. So let's go print f saving model to and we'll put in the path here.
7:27:37
Model save path. I like to print out some things here and there that way. We know what's going on. And I don't need that capital. Why do I? Getting a little bit trigger happy here with the typing. So torch dot save. And we're going to pass in the object parameter here. And if we looked up torch save, we can go. What does this code take? So torch save object f. What is f? A file like object. Okay. Or a string or OS path like object. Beautiful. That's what we've got. A path like
7:28:13
object containing a file name. So let's jump back into here. The object is what? It's our model zero dot state dict. That's what we're saving. And then the file path is model save path. You ready? Let's run this and see what happens. Beautiful. Saving model to models. So it's our model path. And there's our model there. So if we refresh this, what do we have over here? Wonderful. We've saved our trained model. So that means we could potentially if we wanted to,
7:28:44
you could download this file here. That's going to download it from Google CoLab to your local machine. That's one way to do it. But there's also a guide here to save from Google Collaboratory to Google Drive. That way you could use it later on. So there's many different ways. The beauty of pie torches is flexibility. So now we've got a saved model. But let's just check using our LS command. We're going to check models. Yeah, let's just check models. This is going to
7:29:15
check here. So this is list. Wonderful. There's our 01 pie torch workflow model zero dot pth. Now, of course, we've saved a model. How about we try loading it back in and seeing how it works. So if you want to challenge, read ahead on the documentation and try to use torch dot load to bring our model back in. See what happens. I'll see in the next video. Welcome back. In the last video, we wrote some code here to save our pie torch model. I'm just going to exit out of this couple of things
7:29:47
that we don't need just to clear up the screen. And now we've got our dot pth file, because remember dot pth or dot pth is a common convention for saving a pie torch model. We've got it saved there, and we didn't necessarily have to write all of this path style code. But this is just handy for later on if we wanted to functionize this and create it in say a save dot pie file over here, so that we could just call our save function and pass it in a file path where we wanted to save
7:30:16
like a directory and a name, and then it'll save it exactly how we want it for later on. But now we've got a saved model. I issued a challenge of trying to load that model in. So do we have torch dot load in here? Did you try that out? We've got, oh, we've got a few options here. Wonderful. But we're using one of the first ones. So let's go back up here. If we wanted to check the documentation for torch dot load, we've got this option here, load. What happens? Loads
7:30:46
and objects saved with torch dot save from a file. Torch dot load uses Python's unpickling facilities, but treat storages which underlie tenses specially. They are firstly serialized on the CPU, and then I moved the device they were saved from. Wonderful. So this is moved to the device. If later on when we're using a GPU, this is just something to keep in mind. We'll see that when we start to use a CPU and a GPU. But for now, let's practice using the torch dot load method
7:31:17
and see how we can do it. So we'll come back here and we'll go loading a pytorch model. And since we, she's going to start writing here, since we saved our models state debt, so just the dictionary of parameters from a model, rather than the entire model, we'll create a new instance of our model class and load the state deck,
7:31:49
load the saved state deck. That's better state deck into that.
7:31:55
Now, this is just words on a page. Let's see this in action. So to load in a state deck, which is what we say, we didn't save the entire model itself, which is one option. That's extra curriculum, but we saved just the model state deck. So if we remind ourselves what model zero dot state deck looks like, we saved just this. So to load this in, we have to
7:32:20
instantiate a new class or a new instance of our linear regression model class. So to load in a saved state deck, we have to instantiate a new instance of our model class. So let's call this loaded model zero. I like that. That way we can differentiate because it's still going to be the same parameters as model zero, but this way we know that this instance is the loaded version, not just the version we've been training before. So we'll create a new version of it here,
7:32:57
linear regression model. This is just the code that we wrote above, linear regression model. And then we're going to load the saved state deck of model zero. And so this will update the new instance with updated parameters. So let's just check before we load it, we haven't written any code to actually load anything. What does loaded model zero? What does the state deck look like here?
7:33:28
It won't have anything. It'll be initialized with what? Oh, loaded. That's what I called it loaded. See how it's initialized with random parameters. So essentially all we're doing when we load a state dictionary into our new instance of our model is that we're going, hey, take the saved state deck from this model and plug it into this. So let's see what happens when we do that. So loaded model zero. Remember how I said there's
7:33:56
a method to also be aware of up here, which is torch nn module dot load state deck. And because our model is a what, it's a subclass of torch dot nn dot module. So we can call load state deck on our model directly or on our instance. So recall linear regression model is a subclass of nn dot module. So let's call in load state deck. And this is where we call the torch dot load method. And then we pass it the model save path. Is that what we call it? Because torch dot load,
7:34:31
it takes in F. So what's F a file like object or a string or a OS path like object. So that's why we created this path like object up here. Model save path. So all we're doing here, we're creating a new instance, linear regression model, which is a subclass of nn dot module. And then on that instance, we're calling in load state deck of torch dot load model save path. Because what's saved at the model save path, our previous models state deck, which is here.
7:35:04
So if we run this, let's see what happens. All keys match successfully. That is beautiful. And so see the values here, loaded state deck of model zero. Well, let's check the loaded version of that. We now have wonderful, we have the exact same values as above. But there's a little way that we can test this. So how about we go make some predictions. So make some predictions. Just to make sure with our loaded model. So let's put it in a valve mode. Because when you make
7:35:39
predictions, you want it in evaluation mode. So it goes a little bit faster. And we want to also use inference mode. So with torch dot inference mode for making predictions. We want to write this loaded model preds, we're going to make some predictions on the test data as well. So loaded model zero, we're going to forward pass on the X test data. And then we can have a look at the loaded model preds. Wonderful. And then to see if the two models are the same, we can compare
7:36:14
loaded model preds with original model preds. So why preds? These should be equivalent equals equals loaded model preds. Do we have the same thing? False, false, false, what's going on here? Why preds? How much different are they? Oh, where's that happened? Have we made some model preds with this yet? So how about we make some model preds? This is troubleshooting on the fly team. So let's go model zero dot eval. And then with torch dot inference mode,
7:37:00
this is how we can check to see that our two models are actually equivalent. Why preds equals, I have a feeling why preds actually save somewhere else equals model zero. And then we pass it the X test data. And then we might move this above here. And then have a look at what why preds equals. Do we get the same output? Yes, we should. Wonderful. Okay, beautiful. So now we've covered saving and loading models or specifically saving the models state deck. So we saved it here with
7:37:36
this code. And then we loaded it back in with load state deck plus torch load. And then we checked to see by testing equivalents of the predictions of each of our models. So the original one that we trained here, model zero, and the loaded version of it here. So that's saving and loading a model in pytorch. There are a few more things that we could cover. But I'm going to leave that for extra curriculum. We've covered the two main things or three main things. One, two, three.
7:38:04
If you'd like to read more, I'd highly encourage you to go through and read this tutorial here. But with that being said, we've covered a fair bit of ground over the last few videos. How about we do a few videos where we put everything together just to reiterate what we've done. I think that'll be good practice. I'll see you in the next video.
7:38:25
Welcome back. Over the past few videos, we've covered a whole bunch of ground in a pytorch workflow, starting with data, then building a model. Well, we split the data, then we built a model. We looked at the model building essentials. We checked the contents of our model. We made some predictions with a very poor model because it's based off random numbers. We spent a whole bunch of time figuring out how we could train a model. We figured out what the loss function is.
7:38:50
We saw an optimizer. We wrote a training and test loop. We then learned how to save and load a model in pytorch. So now I'd like to spend the next few videos putting all this together. We're not going to spend as much time on each step, but we're just going to have some practice together so that we can reiterate all the things that we've done. So putting it all together, let's go back through the steps above and see it all in one place. Wonderful.
7:39:24
So we're going to start off with 6.1 and we'll go have a look at our workflow. So 6.1 is data,
7:39:35
but we're going to do one step before that. And I'm just going to get rid of this so we have a bit more space. So we've got our data ready. We've turned it into tenses way back at the start. Then we built a model and then we picked a loss function and an optimizer. We built a training loop. We trained our model. We made some predictions. We saw that they were better. We evaluated our model. We didn't use torch metrics, but we got visual. We saw our red dots starting to line up
7:40:00
with the green dots. We haven't really improved through experimentation. We did a little bit of it though, as in we saw that if we trained our model for more epochs, we got better results. So you could argue that we have done a little bit of this, but there are other ways to experiment. We're going to cover those throughout the course. And then we saw how to save and reload a trained model. So we've been through this entire workflow, which is quite exciting, actually.
7:40:24
So now let's go back through it, but we're going to do it a bit quicker than what we've done before, because I believe you've got the skills to do so now. So let's start by importing pytorch.
7:40:36
So you could start the code from here if you wanted to. And that plot live. And actually, if you want, you can pause this video and try to recode all of the steps that we've done by putting some headers here, like data, and then build a model and then train the model, save and load a model, whatever, and try to code it out yourself. If not, feel free to follow along with me and we'll do it together. So import torch from torch import. Oh, would help if I could spell
7:41:05
torch import and n because we've seen that we use an n quite a bit. And we're going to also import map plot live because we like to make some plots because we like to get visual. Visualize visualize visualize as PLT. And we're going to check out pytorch version.
7:41:24
That way we know if you're on an older version, some of the code might not work here. But if you're on a newer version, it should work. If it doesn't, let me know. There we go. 1.10. I'm using 1.10 for this. By the time you watch this video, there may be a later version out. And we're also going to let's create some device agnostic code. So create device agnostic code, because I think we're up to this step now. This means if we've got access to a GPU, our code will use it for potentially
7:41:59
faster computing. If no GPU is available, the code will default to using CPU. We don't necessarily need to use a GPU for our particular problem that we're working on right now because it's a small model and it's a small data set, but it's good practice to write device agnostic code. So that means our code will use a GPU if it's available, or a CPU by default, if a GPU is not available. So set up device agnostic code. We're going to be using a similar setup to this throughout the
7:42:38
entire course from now on. So that's why we're bringing it back. CUDA is available. So remember CUDA is NVIDIA's programming framework for their GPUs, else use CPU. And we're going to print what device are we using? Device. So what we might do is if we ran this, it should be just a CPU for now, right? Yours might be different to this if you've enabled a GPU, but let's change this over to use CUDA. And we can do that if you're using Google Colab, we can change the runtime type
7:43:12
by selecting GPU here. And then I'm going to save this, but what's going to happen is it's going to restart the runtime. So we're going to lose all of the code that we've written above. How can we get it all back? Well, we can go. Run all. This is going to run all of the cells above here. They should all work and it should be quite quick because our model and data aren't too big. And if it all worked, we should have CUDA as our device that we can use here. Wonderful.
7:43:42
So the beauty of Google Colab is that they've given us access to on a video GPU. So thank you, Google Colab. Just once again, I'm paying for the paid version of Google Colab. You don't have to. The free version should give you access to a GPU, or be it it might not be as a later version as GPU as the pro versions give access to. But this will be more than enough for what we're about to recreate. So I feel like that's enough for this video. We've got some device agnostic code ready
7:44:12
to go. And for the next few videos, we're going to be rebuilding this except using device agnostic code. So give it a shot yourself. There's nothing in here that we haven't covered before. So I'll see you in the next video. Let's create some data. Welcome back. In the last video, we set up some device agnostic code and we got ready to start putting everything we've learned together. So now let's continue with that. We're going to recreate some data. Now we could just copy this
7:44:41
code, but we're going to write it out together so we can have some practice creating a dummy data set. And we want to get to about this stage in this video. So we want to have some data that we can plot so that we can build a model to once again, learn on the blue dots to predict the green dots. So we'll come down here data. I'm going to get out of this as well so that we have a bit more room. Let's now create some data using the linear regression formula of y equals weight times
7:45:19
features plus bias. And you may have heard this as y equals mx plus c or mx plus b or something like that, or you can substitute these for different names. Images when I learned this in high school, it was y equals mx plus c. Yours might be slightly different. Yeah, bx plus a. That's what they use here. A whole bunch of different ways to name things, but they're all describing the same thing. So let's see this in code rather than formulaic examples. So we're going to create our weight,
7:45:51
which is 0.7 and a bias, which is 0.3. These are the values we previously used for a challenge you could change these to 0.1 maybe and 0.2. These could be whatever values you'd like to set them as. So weight and bias, the principle is going to be the same thing. We're going to try and build a model to estimate these values. So we're going to start at 0 and we're going to end at 1. So we can just create a straight line and we're going to fill in those between 0 and 1 with a
7:46:24
step of 0.02. And now we'll create the x and y features x and y, which is features and labels actually. So x is our features and y are our labels. x equals torch dot a range and x is a capital Y is that because typically x is a feature matrix. Even though ours is just a vector now, we're going to unsqueeze this so we don't run into dimensionality issues later on. You can check this for yourself without unsqueeze, errors will pop up and y equals weight times
7:47:00
x plus bias. You see how we're going a little bit faster now? This is sort of the pace that we're going to start going for things that we've already covered. If we haven't covered something, we'll slow down, but if we have covered something, I'm going to step it through. We're going to start speeding things up a little. So if we get some values here, wonderful. We've got some x values and they correlate to some y values. We're going to try and use the training values of x to predict
7:47:28
the training values of y and subsequently for the test values. Oh, and speaking of training and test values, how about we split the data? So let's split the data. Split data. So we'll create the train split equals int 0.8. We're going to use 80%, which is where 0.8 comes from, for the length of x. So we use 80% of our samples for the training, which is a typical training and test split, 80, 20. They're abouts. You could use like 70, 30. You could use 90, 10.
7:47:59
It all depends on how much data you have. There's a lot of things in machine learning that are quite flexible. Train split, we're going to index on our data here so that we can create our splits. Google Colab auto corrected my code in a non-helpful way just then. And we're going to do the opposite split for the testing data. Now let's have a look at the lengths of these. If my calculations are correct, we should have about 40 training samples and 10 testing samples. And again, this
7:48:37
may change in the future. When you work with larger data sets, you might have 100,000 training samples and 20,000 testing samples. The ratio will often be quite similar. And then let's plot what's going on here. So plot the data and note, if you don't have the plot predictions function loaded, this will error. So we can just run plot predictions here if we wanted to. And we'll pass it in X train, Y train, X test, Y test. And this should come up with our
7:49:17
plot. Wonderful. So we've just recreated the data that we've been previously using. We've got blue dots to predict green dots. But if this function errors out because you've started the notebook from here, right from this cell, and you've gone down from there, just remember, you'll just have to go up here and copy this function. We don't have to do it because we've run all the cells, but if you haven't run that cell previously, you could put it here and then run it, run it,
7:49:46
and we'll get the same outcome here. Wonderful. So what's next? Well, if we go back to our workflow, we've just created some data. And have we turned it into tenses yet? I think it's just still, oh, yeah, it is. It's tenses because we use PyTorch to create it. But now we're up to building or picking a model. So we've built a model previously. We did that back in build model. So you could refer to that code and try to build a model to fit the data that's going on here. So that's
7:50:16
your challenge for the next video. So building a PyTorch linear model. And why do we call it linear? Because linear refers to a straight line. What's nonlinear? Non-straight. So I'll see you in the next video. Give it a shot before we get there. But we're going to build a PyTorch linear model.
7:50:39
Welcome back. We're going through some steps to recreate everything that we've done. In the last video, we created some dummy data. And we've got a straight line here. So now by the workflow, we're up to building a model or picking a model. In our case, we're going to build one to suit our problem. So we've got some linear data. And I've put building a PyTorch linear model here. I issued you the challenge of giving it a go. You could do exactly the same steps that
7:51:04
we've done in build model. But I'm going to be a little bit cheeky and introduce something new here. And that is the power of torch.nn. So let's see it. What we're going to do is we're going to create a linear model by subclassingnn.module because why a lot of PyTorch models, subclass, and then module. So class linear regression, what should we call this one? Linear regression model v2. How about that? And we'll subclassnn.module. So much similar code to
7:51:41
what we've been writing so far. Or when we first created our linear regression model. And then we're going to put the standard constructor code here, def init underscore underscore. And it's going to take as an argument self. And then we're going to call super dot another underscore init underscore underscore brackets. But we're going to instead of if you recall above back in the build model section, we initialized these parameters ourselves. And I've been hinting
7:52:15
at in the past in videos we've seen before that oftentimes you won't necessarily initialize the parameters yourself. You'll instead initialize layers that have the parameters in built in those layers. We still have to create a forward method. But what we're going to see is how we can use our torch linear layer to do these steps for us. So let's write the code and then we'll step through it. So we'll go usenn.linear because why we're building linear regression model and our data is linear.
7:52:53
And in the past, our previous model has implemented linear regression formula. So for creating the model parameters. So we can go self dot linear layer equals. So this is constructing a variable that this class can use self linear layer equals nn dot linear. Remember, nn in PyTorch stands for neural network. And we have in features as one of the parameters and out features as another parameter. This means we want to take as input of size one and output of size one. Where does that
7:53:35
come from? Well, if we have a look at x train and y train, we have one value of x. Maybe there's too many here. x five will be the first five five and five. So recall, we have one value of x equates to one value of y. So that means within this linear layer, we want to take as one feature x to output one feature y. And we're using just one layer here. So the input and the output shapes of your model in features, out features, what data goes in and what data comes out. These values
7:54:18
will be highly dependent on the data that you're working with. And we're going to see different data or different examples of input features and output features all throughout this course. So but that is what's happening. We have one in feature to one out feature. Now what's happening inside nn.linear. Let's have a look torch and then linear. We go the documentation
7:54:43
applies a linear transformation to the incoming data. Where have we seen this before? y equals x a t plus b. Now they're using different letters, but we've got the same formula as what's happening up here. Look at the same formula as our data. Wait times x plus bias. And then if we look up linear regression formula once again, linear regression formula. We've got this formula here. Now again, these letters can be replaced by whatever letters you like. But this linear layer
7:55:19
is implementing the linear regression formula that we created in our model before. So it's essentially doing this part for us. And behind the scenes, the layer creates these parameters for us. So that's a big piece of the puzzle of pie torch is that as I've said, you won't always be initializing the parameters your model yourself. You'll generally initialize layers. And then you'll use those layers in some Ford computation. So let's see how we could do that. So we've got a linear
7:55:52
layer which takes us in features one and out features one. What should we do now? Well, because we've subclassed nn.module we need to override the Ford method. So we need to tell our model what should it do as the Ford computation. And in here it's going to take itself as input, as well as x, which is conventional for the input data. And then we're just going to return here, self dot linear layer x. Right. And actually, we might use some typing here to say that this
7:56:24
should be a torch tensor. And it's also going to return a torch dot tensor. That's using Python's type ins. So this is just saying, hey, X should be a torch tensor. And I'm going to return you a torch tensor, because I'm going to pass x through the linear layer, which is expecting one in feature and one out feature. And it's going to this linear transform. That's another word for it. Again, pytorch and machine learning in general has many different names of the same thing. I would call
7:56:53
this linear layer. I'm going to write here, also called linear transform, probing layer, fully connected layer, dense layer, intensive flow. So a whole bunch of different names for the same thing, but they're all implementing a linear transform. They're all implementing a version of linear regression y equals x, a ranspose plus b, in features, out features, wonderful. So let's see this in action. So we're going to go set the manual seed so we can
7:57:32
get reproducibility as well, torch dot manual seed. And we're going to set model one equals linear regression. This is model one, because we've already got model zero, linear regression V two, and we're going to check model one, and we're going to check its state dictionary, state dict. There we go. What do we have inside this ordered dict? Has that not created anything for us? Model one, dot state dinked, ordered dink. We haven't got anything here in the regression
7:58:14
model V two. Ideally, this should be outputting a weight and a bias. Yeah, variables, weight, and bias. Let's dig through our code line by line and see what we've got wrong. Ah, did you notice this? The init function so the constructor had the wrong amount of underscores. So it was never actually constructing this linear layer troubleshooting on the fly team. There we go. Beautiful. So we have a linear layer, and we have it is created for us inside a weight and a bias. So effectively,
7:58:50
we've replaced the code we wrote above for build model, initializing a weight and bias parameter with the linear layer. And you might be wondering why the values are slightly different, even though we've used the manual seed. This goes behind the scenes of how PyTorch creates its different layers. It's probably using a different form of randomness to create different types of variables. So just keep that in mind. And to see this in action, we have a conversion here.
7:59:18
So this is what's going on. We've converted, this is our original model class, linear regression. We initialize our model parameters here. We've got a weight and a bias. But instead, we've swapped this in our linear regression model V2. This should be V2 to use linear layer. And then in the forward method, we had to write the formula manually here when we initialize the parameters manually. But because of the power of torch.nn, we have just passed it through the linear layer,
7:59:48
which is going to perform some predefined forward computation in this layer. So this style of what's going on here is how you're going to see the majority of your PyTorch deep learning models created using pre-existing layers from the torch.nn module. So if we go back into torch.nn, torch.nn, we have a lot of different layers here. So we have convolutional layers, pooling layers, padding layers, normalization, recurrent, transformer, linear, we're using a
8:00:19
linear layer, dropout, et cetera, et cetera. So for all of the common layers in deep learning, because that's what neural networks are, they're layers of different mathematical transformations, PyTorch has a lot of pre-built implementations. So that's a little bit of a sneaky trick that I've done to alter our model. But we've still got basically the exact same model as we had before. So what's next? Well, it's to train this model. So let's do that in the next video.
8:00:44
Welcome back. So in the last video, we built a PyTorch linear model, nice and simple using a single nn.linear layer with one in feature, one out feature. And we over read the forward method of nn.module using the linear layer that we created up here. So what's going to happen is when we do the forward parser on our model, we're going to put some data in and it's going to go through this linear layer, which behind the scenes, as we saw with torch and n linear,
8:01:20
behind the scenes, it's going to perform the linear regression formula here. So y equals x, a t plus b. But now case, we've got weight and bias. So let's go back. It's now time to write some training code. But before we do, let's set the model to use the target device. And so in our case, we've got a device of CUDA. But because we've written device agnostic code, if we didn't have access to a CUDA device, a GPU, our default device would be a CPU. So let's check the model
8:01:57
device. We can do that first up here, check the model current device, because we're going to use the GPU here, or we're going to write device agnostic code. That's better to say device agnostic code. That's the proper terminology device. What device are we currently using? This is the CPU, right? So by default, the model will end up on the CPU. But if we set it to model one call dot two device, what do you think it's going to do now? If our current target device is CUDA, we've seen what
8:02:31
two does in the fundamental section, two is going to send the model to the GPU memory. So now let's check whether parameters of our model live dot device. If we send them to the device previously, it was the CPU, it's going to take a little bit longer while the GPU gets fired up and goes, PyTorch goes, Hey, I'm about to send you this model. You ready for it? Boom, there we go. Wonderful. So now our model is on the device or the target device, which is CUDA. And if CUDA wasn't available,
8:03:02
the target device would be CPU. So this would just come out just exactly how we've got it here. But with that being said, now let's get on to some training code. And this is the fun part. What do we have to do? We've already seen this for training. I'm just going to clear up our workspace a little bit here. For training, we need, this is part of the PyTorch workflow, we need a loss function. What does a loss function do? Measures how wrong our model is,
8:03:29
we need an optimizer, we need a training loop and a testing loop. And the optimizer, what does that do? Well, it optimizes the parameters of our model. So in our case, model one dot state dig, what do we have? So we have some parameters here within the linear layer, we have a weight, and we have a bias. The optimizer is going to optimize these random parameters so that they hopefully reduce the loss function, which remember the loss function measures how wrong our model
8:04:01
is. So in our case, because we're working with the regression problem, let's set up the loss function. And by the way, all of these steps are part of the workflow. We've got data ready, we've built or picked a model, we're using a linear model. Now we're up to here 2.1 pick a loss function and an optimizer, we're going to do build a training loop in the same session, because you know what, we're getting pretty darn good at this, loss function equals what?
8:04:27
Well, we're going to use l one loss. So let's set that up and then dot l one loss, which is the same as ma and if we wanted to set up our optimizer, what optimizer could we use? Well, pytorch offers a lot of optimizers in torch dot opt in SGD. That's stochastic gradient descent, because remember gradient descent is the algorithm that optimizes our model parameters. Adam is another popular option. For now, we're going to stick with SGD. LR, which stands for learning rate. In other words,
8:05:01
how big of a step will our optimizer change our parameters with every iteration, a smaller learning rate. So such as 0001 will be a small step. And then a large learning rate, such as 0.1 will be a larger step. Too big of a step. Our model learns too much. And it explodes too small of a step. Our model never learns anything. But oh, we actually have to pass params first. I forgot about that. I got ahead of myself with a learning rate. Params is the parameters we'd like our
8:05:33
optimizer to optimize. So in our case, it's model one dot parameters, because model one is our current target model. Beautiful. So we've got a loss function and an optimizer. Now, let's write a training loop. So I'm going to set torch manual seeds so we can try and get as reproducible as results as possible. Remember, if you get different numbers to what I'm getting, don't worry too much if they're not exactly the same, the direction is more important. So that means if my loss function is
8:06:05
getting smaller, yours should be getting smaller too. Don't worry too much if your fourth decimal place isn't the same as what my values are. So we have a training loop ready to be written here. Epox, how many should we do? Well, we did 200 last time and that worked pretty well. So let's do 200 again. Did you go through the extra curriculum yet? Did you watch the video for the unofficial PyTorch optimization loop song yet? This one here, listen to the unofficial PyTorch optimization
8:06:36
loop song. If not, it's okay. Let's sing it together. So for an epoch in range, epochs, we're going to go through the song in a second. We're going to set the model to train. In our case, it's model one, model to train. Now, step number one is what? Do the forward pass. This is where we calculate the predictions. So we calculate the predictions by passing the training data through our model. And in our case, because the forward method in model one implements the linear layer,
8:07:09
this data is going to go through the linear layer, which is torch.nn.linear and go through the linear regression formula. And then we calculate the loss, which is how wrong our models predictions are. So the loss value equals loss fn. And here we're going to pass in y-pred and y-train.
8:07:31
Then what do we do? We zero the optimizer, optimizer zero grad, which because by default, the optimizer is going to accumulate gradients behind the scenes. So every epoch, we want to reduce those back to zero. So it starts from fresh. We're going to perform back propagation here, back propagation, by calling loss, stop backwards. If the forward pass goes forward through the network, the backward pass goes backwards through the network, calculating the gradients for the
8:08:03
loss function with respect to each parameter in the model. So optimizer step, this next part, is going to look at those gradients and go, you know what? Which way should I optimize the parameters? So because the optimizer is optimizing the model parameters, it's going to look at the loss and go, you know what? I'm going to adjust the weight to be increased. And I'm going to lower the bias and see if that reduces the loss. And then we can do testing. We can do both of these in
8:08:32
the same hit. Now we are moving quite fast through this because we spent a whole bunch of time discussing what's going on here. So for testing, what do we do? We set the model into evaluation mode. That's going to turn off things like dropout and batch normalization layers. We don't have any of that in our model for now, but just it's good practice to always call a vowel whenever you're doing testing. And same with inference mode. We don't need to track gradients and a whole bunch of
8:08:58
other things PyTorch does behind the scenes when we're testing or making predictions. So we use the inference mode context manager. This is where we're going to create test pred, which is going to be our test predictions, because here we're going to pass the test data features, forward pass through our model. And then we can calculate the test loss, which is our loss function. And we're going to compare the test pred to Y test. Wonderful. And then we can print out what's happening.
8:09:28
So what should we print out? How about if epoch divided by 10 equals zero. So every 10 epochs, let's print something out, print. We'll do an F string here, epoch is epoch. And then we'll go loss, which is the training loss, and just be equal to the loss. And then we'll go test loss is equal to test loss. So do you think this will work? It's okay if you're not sure. But let's find out together, hey, oh, we've got a, we need a bracket there. Oh my goodness, what's going on?
8:10:15
Run time error. Expected all tenses to be on the same device. Oh, of course. Do you know what's happening here? But we found at least two devices, CUDA and CPU. Yes, of course, that's what's happened. So what have we done? Up here, we put our model on the GPU. But what's going on here? Our data? Has our data on the GPU? No, it's not. By default, it's on the CPU. So we haven't written device agnostic code for our data. So let's write it here, put data on the target device.
8:10:52
Device agnostic code for data. So remember, one of the biggest issues with pytorch aside from shape errors is that you should have your data or all of the things that you're computing with on the same device. So that's why if we set up device agnostic code for our model, we have to do the same for our data. So now let's put X train to device. Y train equals Y train to device. This is going to create device agnostic code. In our case, it's going to use CUDA because
8:11:25
we have access to a CUDA device. But if we don't, this code will still work. It will still default to CPU. So this is good. I like that we got that error because that's the sum of the things you're going to come across in practice, right? So now let's run this. What's happening here? Hey, look at that. Wonderful. So our loss starts up here nice and high. And then it starts to go right down here for the training data. And then the same for the testing data. Beautiful.
8:11:56
Right up here. And then all the way down. Okay. So this looks pretty good on the test data set. So how can we check this? How can we evaluate our model? Well, one way is to check its state deck. So state decked. What do we got here? What are our weight and bias? Oh my gosh, so close. So we just set weight and bias before to be 0.7 and 0.3. So this is what our model has estimated our parameters to be based on the training data. 0.6968. That's pretty close to 0.7,
8:12:31
nearly perfect. And the same thing with the bias 0.3025 versus the perfect value is 0.93. But remember, in practice, you won't necessarily know what the ideal parameters are. This is just to exemplify what our model is doing behind the scenes. It's moving towards some ideal representative parameters of whatever data we're working with. So in the next video, I'd like you to give it a go of before we get to the next video, make some predictions with our model and plot them on the
8:13:02
original data. How close to the green dots match up with the red dots? And you can use this plot predictions formula or function that we've been using in the past. So give that a go and I'll see you in the next video. But congratulations. Look how quickly we just trained a model using the steps that we've covered in a bunch of videos so far and device agnostic code. So good. I'll see you soon. In the last video, we did something very, very exciting. We worked through
8:13:33
training an entire neural network. Some of these steps took us an hour or so worth of videos to go back through before. But we coded that in one video. So you're ready listening the song just to remind ourselves of what's going on. For an epoch in a range, call model dot train, do the forward pass, calculate the loss, optimizer zero grad, loss backward, optimizer step, step, step, let's test, come on a dot eval with torch inference mode, do the forward pass, calculate the loss,
8:14:06
print out what's happening. And then we do it again, again, again, for another epoch in a range. Now I'm kidding. We'll just leave it there. We'll just leave it there. But that's the unofficial pytorch optimization loop song. We created some device agnostic code so that we could make the calculations on the same device as what our model is because the models also using device agnostic code. And so now we've got to evaluate our models. We've looked at the loss and the test
8:14:36
lost here. And we know that our models loss is going down. But what does this actually equate to when it makes predictions? That's what we're most interested in, right? And we've looked at the parameters. They're pretty close to the ideal parameters. So at the end of last video, I issued you the challenge to making and evaluating predictions to make some predictions and plot them. I hope you gave it a shot. Let's see what it looks like together. Hey, so turn the model into evaluation
8:15:08
mode. Why? Because every time we're making predictions or inference, we want our model to be in a vowel mode. And every time we're training, we want our model to be in training mode. And then we're going to make predictions on the test data, because we train on the train data, and we evaluate our model on the test data data that our model has never actually seen, except for when it makes predictions. With torch inference mode, we turn on inference mode whenever we make inference or
8:15:37
predictions. So we're going to set Y threads equal to model one, and the test data goes in here. Let's have a look at what the Y threads look like. Wonderful. So we've got a tensor here. It shows us that they're still on the device CUDA. Why is that? Well, that's because previously we set the model one to the device, the target device, the same with the test data. So subsequently, our predictions are also on the CUDA device. Now, let's bring in the plot predictions function here.
8:16:09
So check out our model predictions visually. We're going to adhere to the data explorer's motto of visualize visualize visualize plot predictions. And predictions are going to be set to equals Y threads. And let's have a look. How good do these look? Oh, no.
8:16:35
Oh, we've got another error type error. Can't convert CUDA device type tensor to NumPy. Oh, of course. Look what we've done. So our plot predictions function, if we go back up, where did we define that? What does our plot predictions function use? It uses matplotlib, of course, and matplotlib works with NumPy, not pytorch. And NumPy is CPU based. So of course, we're running into another error down here, because we just said that our predictions are on the CUDA
8:17:07
device. They're not on the CPU. They're on a GPU. So it's giving us this helpful information here. Use tensor dot CPU to copy the tensor to host memory first. So this is our tensor. Let's call dot CPU and see what happens then. Is that going to go to CPU? Oh, my goodness. Look at that. Look at that. Go the linear layer. The red dots, the predictions are basically on top of the testing data. That is very exciting. Now again, you may not get the exact same numbers here, and that is
8:17:38
perfectly fine. But the direction should be quite similar. So your red dots should be basically on top of the green dots, if not very slightly off. But that's okay. That's okay. We just want to focus on the direction here. So thanks to the power of back propagation here and gradient descent, our models random parameters have updated themselves to be as close as possible to the ideal parameters. And now the predictions are looking pretty darn good for what we're trying to predict.
8:18:09
But we're not finished there. We've just finished training this model. What would happen if our notebook disconnected right now? Well, that wouldn't be ideal, would it? So in the next part, we're going to move on to 6.5, saving, and loading a trained model. So I'm going to give you a challenge here as well, is to go ahead and go back and refer to this code here, saving model in PyTorch, loading a PyTorch model, and see if you can save model one, the state dictionary of
8:18:39
model one, and load it back in and get something similar to this. Give that a shot, and I'll see you in the next video. Welcome back. In the last video, we saw the power of the torch.nn.linear layer, and back propagation and gradient descent. And we've got some pretty darn good predictions out of our model. So that's very exciting. Congratulations. You've now trained two machine learning models. But it's not over yet. We've got to save and load our trained model. So I
8:19:10
issued you the challenge in the last video to try and save and load the model yourself. I hope you gave that a go. But we're going to do that together in this video. So we're going to start by importing path because we would like a file path to save our model to. And the first step we're going to do is create models directory. We don't have to recreate this because I believe we already have one. But I'm going to put the code here just for completeness. And this is just so if you
8:19:37
didn't have a models directory, this would create one. So model path is going to go to path
8:19:48
models. And then we'd like to model path dot maker, we're going to call maker for make directory. We'll set parents equal to true. And if it exists, okay, that'll also be true. So we won't get an error. Oh my gosh, Google collab. I didn't want that. We won't get an error if it already exists. And two, we're going to create a model save path. So if you recall that pytorch objects in general have the extension of what? There's a little pop quiz before we get to the end of this sentence.
8:20:21
So this is going to be pytorch workflow for this module that we're going through. This one here, chapter 01 pytorch workflow model one. And they usually have the extension dot PT for pytorch or PT H for pytorch as well. I like PT H. But just remember, sometimes you might come across slightly different versions of that PT or PT H. And we're going to create the model save name or the save path. It's probably a better way to do it is going to be model path. And then we can use because we're
8:20:55
using the path lib module from Python, we can save it under model name. And so if we look at this, what do we get model save path? We should get Oh, path is not defined. Oh, too many capitals here, Daniel. The reason why I'm doing these in capitals is because oftentimes hyper parameters such as epochs in machine learning are set as hyper parameters LR could be learning rate. And then you could have
8:21:25
as well model name equals Yeah, yeah, yeah. But that's just a little bit of nomenclature trivia for later on. And model save path, we've done that. Now we're going to save the model state dictionary rather than the whole model, save the model state deck, which you will find the pros and cons of in where in the pytorch documentation for saving and loading model, which was a little bit of extra curriculum for a previous video. But let's have a look at our model save path will print it out.
8:21:58
And we'll go torch save, we'll set the object that we're trying to save to equal model one dot state deck, which is going to contain our trained model parameters. We can inspect what's going on in here, state deck. They'll show us our model parameters. Remember, because we're only using a single linear layer, we only have two parameters. But in practice, when you use a model with maybe hundreds of layers or tens of millions of parameters, viewing the state deck explicitly, like we are now,
8:22:30
might not be too viable of an option. But the principle still remains a state deck contains all of the models trained or associated parameters, and what state they're in. And the file path we're going to use is, of course, the model save path, which we've seen here is a POSIX path. Let's save our model. Wonderful saving model to this file path here. And if we have a look at our folder, we should have two saved models now, beautiful to save models. This one for us from the workflow
8:23:02
we did before up here, saving a model in PyTorch, loading a PyTorch model. And now the one we've got, of course, model one is the one that we've just saved. Beautiful. So now let's load a model. We're going to do both of these in one video. Load a PyTorch model. You know what, because we've had a little bit of practice so far, and we're going to pick up the pace. So let's go loaded, let's call it, we'll create a new instance of loaded model one, which is, of course, our linear regression model
8:23:31
V2, which is the version two of our linear regression model class, which subclasses, what? Subclasses and n.module. So if we go back here up here to where we created it. So linear regression model V2 uses a linear layer rather than the previous iteration of linear regression model, which we created right up here. If we go up to here, which explicitly defined the parameters, and then implemented a linear regression formula in the forward method, the difference between
8:24:03
what we've got now is we use PyTorch's pre-built linear layer, and then we call that linear layer in the forward method, which is probably the far more popular way of building PyTorch models, is stacking together pre-built NN layers, and then calling them in some way in the forward method. So let's load it in. So we'll create a new instance of linear regression model V2, and now what do we do? We've created a new instance, I'm just going to get out of this, make some space for us.
8:24:38
We want to load the model state deck, the saved model one state deck, which is the state deck that we just saved beforehand. So we can do this by going loaded model one, calling the load state decked method, and then passing it torch dot load, and then the file path of where we saved that PyTorch object before. But the reason why we use the path lib is so that we can just call model save path in here. Wonderful. And then let's check out what's going on. Or actually, we need to
8:25:13
put the target model or the loaded model to the device. The reason being is because we're doing all our computing with device agnostic code. So let's send it to the device. And I think that'll be about it. Let's see if this works. Oh, there we go. Linear regression model V2 in features one, out features one, bias equals true. Wonderful. Let's check those parameters. Hey, next loaded model one dot parameters. Are they on the right device? Let's have a look. Beautiful. And let's just check
8:25:51
the loaded state dictionary of loaded model one. Do we have the same values as we had previously? Yes, we do. Okay. So to conclusively make sure what's going on, let's evaluate the loaded model. Evaluate loaded model, loaded model one. What do we do for making predictions? Or what do we do to evaluate? We call dot a vowel. And then if we're going to make some predictions, we use torch inference mode with torch inference mode. And then let's create loaded model one, threads
8:26:24
equals loaded model one. And we'll pass it the test data. And now let's check for a quality between Y threads, which is our previous model one preds that we made up here, Y threads. And we're going to compare them to the fresh loaded model one preds. And should they be the same?
8:26:50
Yes, they are beautiful. And we can see that they're both on the device CUDA. How amazing is that? So I want to give you a big congratulations, because you've come such a long way. We've gone through the entire PyTorch workflow from making data, preparing and loading it to building a model. All of the steps that come in building a model, there's a whole bunch there, making predictions, training a model, we spent a lot of time going through the training steps. But trust me, it's
8:27:18
worth it, because we're going to be using these exact steps all throughout the course. And in fact, you're going to be using these exact steps when you build PyTorch models after this course. And then we looked at how to save a model so we don't lose all our work, we looked at loading a model, and then we put it all together using the exact same problem, but in far less time. And as you'll see later on, we can actually make this even quicker by functionalizing some of the code we've already
8:27:42
written. But I'm going to save that for later. I'll see you in the next video, where I'm just going to show you where you can find some exercises and all of the extra curriculum I've been talking about throughout this section 01 PyTorch workflow. I'll see you there. Welcome back. In the last video, we finished up putting things together by saving and loading our trained model, which is super exciting, because let's come to the end of the PyTorch workflow section. So now, this section
8:28:12
is going to be exercises and extra curriculum, or better yet, where you can find them. So I'm going to turn this into markdown. And I'm going to write here for exercises and extra curriculum.
8:28:27
Refer to. So within the book version of the course materials, which is at learnpytorch.io, we're in the 01 section PyTorch workflow fundamentals. There'll be more here by the time you watch this video likely. And then if we go down here, at the end of each of these sections, we've got the table of contents over here. We've got exercises and extra curriculum. I listed a bunch of things throughout this series of 01 videos, like what's gradient descent and what's back propagation. So
8:28:58
I've got plenty of resources to learn more on that. There's the loading and saving PyTorch documentation. There's the PyTorch cheat sheet. There's a great article by Jeremy Howard for a deeper understanding of what's going on in torch.nn. And there's, of course, the unofficial PyTorch optimization loop song by yours truly, which is a bit of fun. And here's some exercises. So the exercises here are all based on the code that we wrote throughout section 01. So there's
8:29:27
nothing in the exercises that we haven't exactly covered. And if so, I'll be sure to put a note in the exercise itself. But we've got create a straight line data set using the linear regression formula. And then build a model by subclassing and end up module. So for these exercises, there's an exercise notebook template, which is, of course, linked here. And in the PyTorch deep learning GitHub, if we go into here, and then if we go into extras, and if we go into exercises, you'll
8:29:56
find all of these templates here. They're numbered by the same section that we're in. This is PyTorch workflow exercises. So if you wanted to complete these exercises, you could click this notebook here, open in Google CoLab. I'll just wait for this to load. There we go. And you can start to write some code here. You could save a copy of this in your own Google Drive and go through this. It's got some notes here on what you should be doing. You can, of course, refer to the text-based
8:30:26
version of them. They're all here. And then if you want an example of what some solutions look like, now, please, I can't stress enough that I would highly, highly recommend trying the exercises yourself. You can use the book that we've got here. This is just all the code from the videos. You can use this. You can use, I've got so many notebooks here now, you can use all of the code that we've written here to try and complete the exercises. But please give them a go yourself.
8:30:52
And then if you go back into the extras folder, you'll also find solutions. And this is just one example solutions for section 01. But I'm going to get out of that so you can't cheat and look at the solutions first. But there's a whole bunch of extra resources all contained within the PyTorch deep loaning repo, extras, exercises, solutions, and they're also in the book version of the course. So I'm just going to link this in here. I'm going to put this right at the bottom
8:31:21
here. Wonderful. But that is it. That is the end of the section 01 PyTorch workflow. So exciting. We went through basically all of the steps in a PyTorch workflow, getting data ready, turning into tenses, build or pick a model, picking a loss function on an optimizer. We built a training loop. We fit the model to the data. We made a prediction. We evaluated our model. We improved through experimentation by training for more epochs.
8:31:47
We'll do more of this later on. And we saved and reload our trained model. But that's going to finish 01. I will see you in the next section. Friends, welcome back. We've got another very exciting module. You ready? Neural network classification with
8:32:10
PyTorch. Now combining this module once we get to the end with the last one, which was regression. So remember classification is predicting a thing, but we're going to see this in a second. And regression is predicting a number. Once we've covered this, we've covered two of the the biggest problems in machine learning, predicting a number or predicting a thing. So let's start off with before we get into any ideas or code, where can you get help?
8:32:38
First things first is follow along with the code. If you can, if in doubt, run the code. Try it for yourself. Write the code. I can't stress how important this is. If you're still stuck, press shift, command, and space to read the doc string of any of the functions that we're running. If you are on Windows, it might be control. I'm on a Mac, so I put command here. If you're still stuck, search for your problem. If an error comes up, just copy and paste
8:33:04
that into Google. That's what I do. You might come across resources like Stack Overflow or, of course, the PyTorch documentation. We'll be referring to this a lot again throughout this section. And then finally, oh wait, if you're still stuck, try again. If in doubt, run the code. And then finally, if you're still stuck, don't forget, you can ask a question. The best place to do so will be on the course GitHub, which will be at the discussions page, which is linked here.
8:33:32
If we load this up, there's nothing here yet, because as I record these videos, the course hasn't launched yet, but press new discussion. Talk about what you've got. Problem with XYZ. Let's go ahead. Leave a video number here and a timestamp, and that way, we'll be able to help you out as best as possible. So video number, timestamp, and then your question here, and you can select Q&A. Finally, don't forget that this notebook that we're about to go through is based
8:34:01
on chapter two of the Zero to Mastery Learn PyTorch for deep learning, which is neural network classification with PyTorch. All of the text-based code that we're about to write is here. That was a little spoiler. And don't forget, this is the home page. So my GitHub repo slash PyTorch deep learning for all of the course materials, everything you need will be here. So that's very important. How can you get help? But this is the number one. Follow along with the code and try
8:34:28
to write it yourself. Well, with that being said, when we're talking about classification, what is a classification problem? Now, as I said, classification is one of the main problems of machine learning. So you probably already deal with classification problems or machine learning powered classification problems every day. So let's have a look at some examples. Is this email spam or not spam? Did you check your emails this morning or last night or whenever? So chances are
8:34:57
that there was some sort of machine learning model behind the scenes. It may have been a neural network. It may have not that decided that some of your emails won't spam. So to Daniel, at mrdberg.com, hey, Daniel, this steep learning course is incredible. I can't wait to use what I've learned. Oh, that's such a nice message. If you want to send that email directly to me, you can. That's my actual email address. But if you want to send me this email, well, hopefully
8:35:21
my email, which is hosted by some email service detects this as spam because although that is a lot of money and it would be very nice, I think if someone can't spell too well, are they really going to pay me this much money? So thank you email provider for classifying this as spam. And now because this is one thing or another, not spam or spam, this is binary classification. So in this case, it might be one here and this is a zero or zero or one. So one thing or another, that's binary
8:35:51
classification. If you can split it into one thing or another, binary classification. And then we have an example of say we had the question, we asked our photos app on our smartphone or whatever device you're using. Is this photo of sushi steak or pizza? We wanted to search our photos for every time we've eaten sushi or every time we've eaten steak or every time we've eaten pizza far out and this looks delicious. But this is multi class classification. Now, why is this? Because we've
8:36:21
got more than two things. We've got 123. And now this could be 10 different foods. It could be 100 different foods. It could be 1000 different categories. So the image net data set, which is a popular data set for computer vision, image net, we go to here, does it say 1000 anywhere, 1k or 1000?
8:36:43
No, it doesn't. But if we go image net 1k, download image net data, maybe it's here. It won't say it, but you just, oh, there we go, 1000 object classes. So this is multi class classification because it has 1000 classes, that's a lot, right? So that's multi class classification, more than one thing or another. And finally, we might have multi label classification, which is what tags should this article have when I first got into machine learning, I got these
8:37:17
two mixed up a whole bunch of times. Multi class classification has multiple classes such as sushi steak pizza, but assigns one label to each. So this photo would be sushi in an ideal world. This is steak and this is pizza. So one label to each. Whereas multi label classification means you could have multiple different classes. But each of your target samples such as this Wikipedia article, what tags should this article have? It may have more than one label. It might have three labels,
8:37:47
it might have 10 labels. In fact, what if we went to the Wikipedia page for deep learning Wikipedia and does it have any labels? Oh, there we go. Where was that? I mean, you can try this yourself. This is just the Wikipedia page for deep learning. There is a lot, there we go categories deep learning, artificial neural networks, artificial intelligence and emerging technologies. So that is an example. If we wanted to build a machine learning model to say, read all of the text in
8:38:15
here and then go tell me what are the most relevant categories to this article? It might come up with something like these. In this case, because it has one, two, three, four, it has multiple labels rather than just one label of deep learning, it could be multi label classification. So we'll go back. But there's a few more. These will get you quite far in the world of classification. So let's dig a little deeper on binary versus multi class classification. You may have already
8:38:45
experienced this. So in my case, if I search on my phone in the photos app for photos of a dog, it might come here. If I search for photos of a cat, it might come up with this. But if I wanted to train an algorithm to detect the difference between photos of these are my two dogs. Aren't they cute? They're nice and tired and they're sleeping like a person. This is seven. Number seven, that's her name. And this is Bella. This is a cat that me and my partner rescued.
8:39:11
And so I'm not sure what this cat's name is actually. So I'd love to give it a name, but I can't. So binary classification, if we wanted to build an algorithm, we wanted to feed it, say, 10,000 photos of dogs and 10,000 photos of cats. And then we wanted to find a random image on the internet and pass it through to our model and say, hey, is this a dog or is this a cat? It would be binary classification because the options are one thing or another dog or cat. But then for
8:39:39
multi-class classification, let's say we've been working on a farm and we've been taking some photos of chickens because they groovy, right? Well, we updated our model and added some chicken photos in there. We would now be working with a multi-class classification problem because we've got more than one thing or another. So let's jump in to what we're going to cover. This is broadly, by the way, because this is just text on a page. You know, I like to just write code of what we're
8:40:06
actually doing. So we're going to look at the architecture of a neural network classification model. We're going to check what the input shapes and output shapes of a classification model are features and labels. In other words, because remember, machine learning models, neural networks love to have numerical inputs. And those numerical inputs often come in tenses. Tenses have different shapes, depending on what data you're working with. We're going to see all of this in code, creating
8:40:32
custom data to view, fit and predict on. We're going to go back through our steps in modeling. We covered this a fair bit in the previous section, but creating a model for neural network classification. It's a little bit different to what we've done, but not too out landishly different. We're going to see how we can set up a loss function and an optimizer for a classification model. We'll recreate a training loop and a evaluating loop or a testing loop. We'll see how we can save and
8:40:56
load our models. We'll harness the power of nonlinearity. Well, what does that even mean? Well, if you think of what a linear line is, what is that? It's a straight line. So you might be able to guess what a nonlinear line looks like. And then we'll look at different classification evaluation methods. So ways that we can evaluate our classification models. And how are we going to do all of this? Well, of course, we're going to be part cook, part chemist, part artist, part
8:41:24
science. But for me, I personally prefer the cook side of things because we're going to be cooking up lots of code. So in the next video, before we get into coding, let's do a little bit more on what are some classification inputs and outputs. I'll see you there. Welcome back. In the last video, we had a little bit of a brief overview of what a classification problem is. But now, let's start to get more hands on by discussing what the actual inputs to a classification problem
8:41:53
look like and the outputs look like. And so let's say we had our beautiful food photos from before, and we were trying to build this app here called maybe food vision to understand what foods are in the photos that we take. And so what might this look like? Well, let's break it down to inputs, some kind of machine learning algorithm, and then outputs. In this case, the inputs we want to numerically represent these images in some way, shape or form. Then we want
8:42:24
to build a machine learning algorithm. Hey, one might actually exist. We're going to see this later on in the transfer learning section for our problem. And then we want some sort of outputs. And in the case of food vision, we want to know, okay, this is a photo of sushi. And this is a photo of steak. And this is a photo of pizza. You could get more hands on and technical and complicated, but we're just going to stick with single label multi class classification. So it could be a sushi photo,
8:42:52
it could be a steak photo, or it could be a pizza photo. So how might we numerically represent these photos? Well, let's just say we had a function in our app that every photo that gets taken automatically gets resized into a square into 224 width and 224 height. This is actually quite a common dimensionality for computer vision problems. And so we've got the width dimension, we've got the height, and then we've got this C here, which isn't immediately recognizable. But in the case
8:43:23
of pictures, they often get represented by width, height color channels. And the color channels is red, green and blue, which is each pixel in this image has some value of red, green or blue, that makes whatever color is displayed here. And this is one way that we can numerically represent an image by taking its width, its height and color channels, and whatever number makes up this particular image. We're going to see this later on when we work with computer vision problems.
8:43:55
So we create a numerical encoding, which is the pixel values here. Then we import the pixel values of each of these images into a machine learning algorithm, which is often already exists. And if it doesn't exist for our particular problem, hey, well, we're learning the skills to build them now, we could use pytorch to build a machine learning algorithm for this. And then outputs, what might these look like? Well, in this case, these are prediction probabilities, which the outputs of
8:44:23
machine learning models are never actually discrete, which means it is definitely pizza. It will give some sort of probability value between zero and one for say the closer to one, the more confident our model is that it's going to be pizza. And the closer to zero is means that, hey, this photo of pizza, let's say this one, and we're trying to predict sushi. Well, it doesn't think that it's sushi. So it's giving it quite a low value here. And then the same for
8:44:53
steak, but it's really high, the value here for pizza. We're going to see this hands on. And then it's the opposite here. So it might have got this one wrong. But with more training and more data, we could probably improve this prediction. That's the whole idea of machine learning, is that if you adjust the algorithm, if you adjust the data, you can improve your predictions. And so
8:45:16
the ideal outputs that we have here, this is what our models going to output. But for our case of building out food vision, we want to bring them back to. So we could just put all of these numbers on the screen here, but that's not really going to help people. We want to put out labels of what's going on here. So we can write code to transfer these prediction probabilities into these labels too. And so how did these labels come about? How do these predictions come about? Well,
8:45:44
it comes from looking at lots of different samples. So this loop, we could keep going, improve these, find the ones where it's wrong, add more images here, train the model again, and then make our app better. And so if we want to look at this from a shape perspective,
8:46:01
we want to create some tenses for an image classification example. So we're building food vision.
8:46:08
We've got an image again, this is just reiterating on some of the things that we've discussed. We've got a width of 224 and a height of 224. This could be different. This could be 300, 300. This could be whatever values that you decide to use. Then we numerically encoded in some way, shape or form. We use this as the inputs to our machine learning algorithm, because of what? Computers and machine learning algorithms, they love numbers. They can find patterns in here
8:46:31
that we couldn't necessarily find. Or maybe we could, if you had a long enough time, but I'd rather write an algorithm to do it for me. Then it has some outputs, which comes in the formal prediction probabilities, the closer to one, the more confident model is and saying, hey, I'm pretty damn confident that this is a photo of sushi. I don't think it's a photo of steak. So I'm giving that zero. It might be a photo of pizza, but I don't really think so.
8:46:55
So I'm giving it quite a low prediction probability. And so if we have a look at what the shapes are for our tenses here, if this doesn't make sense, don't worry. We're going to see the code to do all of this later on. But for now, we're just focusing on a classification input and output. The big takeaway from here is numerical encoding, outputs and numerical encoding. But we want to change these numerical codings from the outputs to something that we understand, say the word sushi.
8:47:22
But this tensor may be batch size. We haven't seen what batch size is. That's all right. We're going to cover it. Color channels with height. So this is represented as a tensor of dimensions. It could be none here. None is a typical value for a batch size, which means it's blank. So when we use our model and we train it, all the code that we write with pytorch will fill in this behind the scenes. And then we have three here, which is color channels. And we have 224, which is the width.
8:47:50
And we have 224 as well, which is the height. Now there is some debate in the field on the ordering. We're using an image as our particular example here on the ordering of these shapes. So say, for example, you might have height width color channels, typically width and height come together in this order. Or they're just side by side in the tensor in terms of their whether dimension appears. But color channels sometimes comes first. That means after the batch size or at the end here.
8:48:17
But pytorch, the default for now is color channels with height, though you can write code to change this order because tenses are quite flexible. And so or the shape could be 32 for the batch size, three, two, two, four, two, two, four, because 32 is a very common batch size. And you don't believe me? Well, let's go here. Yarn LeCoon 32 batch size. Now what is a batch size? Great tweet. Just keep
8:48:46
this in mind for later on. Training with large mini batches is bad for your health. More importantly, it's bad for your test error. Friends don't let friends use mini batches larger than 32. So this is quite an old tweet. However, it still stands quite true. Because like today, it's 2022 when I'm recording these videos, there are batch sizes a lot larger than 32. But 32 works pretty darn well for a lot of problems. And so this means that if we go back to our slide, that if we use
8:49:16
a batch size of 32, our machine learning algorithm looks at 32 images at a time. Now why does it do this? Well, because sadly, our computers don't have infinite compute power. In an ideal world, we look at thousands of images at a time, but it turns out that using a multiple of eight here is actually quite efficient. And so if we have a look at the output shape here, why is it three? Well, because we're working with three different classes, one, two, three. So we've got shape equals
8:49:45
three. Now, of course, as you could imagine, these might change depending on the problem you're working with. So say if we just wanted to predict if a photo was a cat or a dog, we still might have this same representation here because this is the image representation. However, the shape here may be two, or will be two because it's cat or dog, rather than three classes here, but a little bit confusing as well with binary classification, you could have the shape just being one here.
8:50:13
But we're going to see this all hands on. Just remember, the shapes vary with whatever problem you're working on. The principle of encoding your data as a numerical representation stays the same for the inputs. And the outputs will often be some form of prediction probability based on whatever class you're working with. So in the next video, right before we get into coding, let's just discuss the high level architecture of a classification model. And remember, architecture is just like
8:50:45
the schematic of what a neural network is. I'll see you there. Welcome back. In the last video, we saw some example classification inputs and outputs. The main takeaway that the inputs to a classification model, particularly a neural network, want to be some form of numerical representation. And the outputs are often some form of prediction probability. So let's discuss the typical architecture of a classification model. And hey, this is just going to be text
8:51:13
on a page, but we're going to be building a fair few of these. So we've got some hyper parameters over here. We've got binary classification. And we've got multi class classification. Now, there are some similarities between the two in terms of what problem we're working with. But there also are some differences here. And by the way, this has all come from, if we go to the book version of the course, we've got what is a classification problem. And we've got
8:51:41
architecture of a classification neural network. So all of this text is available at learnpytorch.io and in section two. So we come back. So the input layer shape, which is typically decided by the parameter in features, as you can see here, is the same of number of features. So if we were working on a problem, such as we brought it to predict whether someone had heart disease or not, we might have five input features, such as one for age, a number for age,
8:52:09
it might be in my case, 28, sex could be male, height, 180 centimeters. If I've been growing overnight, it's really close to 177. Wait, well, it depends on how much I've eaten, but it's around about 75 kilos and smoking status, which is zero. So it could be zero or one, because remember, we want numerical representation. So for sex, it could be zero for males, one for female, height could be its number, weight could be its number as well. All of these numbers could be
8:52:37
more, could be less as well. So this is really flexible. And it's a hyper parameter. Why? Because we decide the values for each of these. So in the case of our image prediction problem, we could have in features equals three for number of color channels. And then we go
8:52:54
hidden layers. So there's the blue circle here. I forgot that this was all timed and colorful. But let's just discuss hidden layers. Each of these is a layer and n dot linear and n dot linear and n dot relu and n dot linear. So that's the kind of the syntax you'll see in PyTorch for a layer is nn dot something. Now, there are many different types of layers in this in PyTorch. If we go torch and n, basically everything in here is a layer in a neural network. And then if we
8:53:22
look up what a neural network looks like, neural network, recall that all of these are different layers of some kind of mathematical operation. Input layer, hidden layer, you could have as many hidden layers as you want. Do we have ResNet architecture? The ResNet architecture, some of them have 50 layers. Look at this. Each one of these is a layer. And this is only the 34 layer version. I mean, there's ResNet 152, which is 152 layers. We're not at that yet.
8:53:56
But we're working up the tools to get to that stage. Let's come back to here. The neurons per hidden layer. So we've got these, out features, the green circle, the green square. Now, this is, if we go back to our neural network picture, this is these. Each one of these little things is a neuron, some sort of parameter. So if we had 100, what would that look like? Well, we'd have a fairly big graphic. So this is why I like to teach with code because you could customize
8:54:30
this as flexible as you want. So behind the scenes, PyTorch is going to create 100 of these little circles for us. And within each circle is what? Some sort of mathematical operation. So if we come back, what do we got next? Output layer shape. So this is how many output features we have. So in the case of binary classification is one, one class or the other. We're going to see this later on. Multi-class classification is you might have three output features,
8:54:56
one per class, e.g., one for food, person or dog, if you're building a food, person or dog, image classification model. Hidden layer activation, which is, we haven't seen these yet. Relu, which is a rectified linear unit, but can be many others because PyTorch, of course, has what? Has a lot of non-linear activations. We're going to see this later on. Remember, I'm kind of planting the seed here. We've seen what a linear line is, but I want you to imagine what a non-linear line is.
8:55:25
It's going to be a bit of a superpower for our classification problem. What else do we have? Output activation. We haven't got that here, but we'll also see this later on, which could be sigmoid for, which is generally sigmoid for binary classification, but softmax for multi-class classification. A lot of these things are just names on a page. We haven't seen them yet. I like to teach them as we see them, but this is just a general overview of what we're going to
8:55:49
cover. Loss function. What loss function or what does a loss function do? It measures how wrong our model's predictions are compared to what the ideal predictions are. So for binary classification, we might use binary cross entropy loss in PyTorch, and for multi-class classification, we might just use cross entropy rather than binary cross entropy. Get it? Binary classification? Binary cross entropy? And then optimizer. SGD is stochastic gradient descent.
8:56:18
We've seen that one before. Another common option is the atom optimizer, and of course, the torch.optim package has plenty more options. So this is an example multi-class classification problem. This network here. Why is that? And we haven't actually seen an end up sequential, but as you could imagine, sequential stands for it just goes through each of these steps. So multi-class classification, because it has three output features, more than one thing or
8:56:47
another. So three for food, person or dog, but going back to our food vision problem, we could have the input as sushi, steak, or pizza. So we've got three output features, which would be one prediction probability per class of image. We have three classes, sushi, steak, or pizza. Now, I think we've done enough talking here, and enough just pointing to text on slides. How about in the next video? Let's code. I'll see you in Google CoLab.
8:57:22
Welcome back. Now, we've done enough theory of what a classification problem is, what the inputs and outputs are and the typical architecture. Let's get in and write some code. So I'm going to get out of this, and going to go to colab.research.google.com, so we can start writing some PyTorch code. I'm going to click new notebook. We're going to start exactly from scratch. I'm going to name this section two, and let's call it neural network classification with PyTorch. I'm going to put
8:57:58
underscore video, because I'll just show you, you'll see this in the GitHub repo. But for all the video notebooks, the ones that I write code during these videos that you're watching, the exact code is going to be saved on the GitHub repo under video notebooks. So there's 00, which is the fundamentals, and there's the workflow underscore video. But the reference notebook with all the pretty pictures and stuff is in the main folder here. So PyTorch classification that I pi and b
8:58:26
are actually, maybe we'll just rename it that PyTorch classification. But we know it's with neural networks. PyTorch classification. Okay, and let's go here. We'll add a nice title. So O2, neural network classification with PyTorch. And so we'll remind ourselves, classification is a problem of predicting whether something is one thing or another. And there can be multiple
8:59:02
things as the options, such as email, spam or not spam, photos of dogs or cats or pizza or sushi or steak. Lots of talk about food. And then I'm just going to link in here, this resource, because this is the book version of the course. These are what the videos are based off. So book version of this notebook. And then all the resources are in here. All other resources in the GitHub, and then stuck. Ask a question here, which is under the discussions tab. We'll
8:59:46
copy that in here. That way we've got everything linked and ready to go. But as always, what's our first step in our workflow? This is a little test. See if you remember. Well, it's data, of course, because all machine learning problems start with some form of data. We can't write a machine learning algorithm to learn patterns and data that doesn't exist. So let's do this video. We're going to make some data. Of course, you might start with some of your own that exists. But for now,
9:00:14
we're going to focus on just the concepts around the workflow. So we're going to make our own custom data set. And to do so, I'll write the code first, and then I'll show you where I get it from. We're going to import the scikit loan library. One of the beautiful things about Google Colab is that it has scikit loan available. You're not sure what scikit loan is. It's a very popular machine learning library. PyTorch is mainly focused on deep learning, but scikit loan is
9:00:42
focused on a lot of things around machine learning. So Google Colab, thank you for having scikit loan already installed for us. But we're going to import the make circles data set. And rather than talk about what it does, let's see what it does. So make 1000 samples. We're going to go N samples equals 1000. And we're going to create circles. You might be wondering why circles. Well, we're going to see exactly why circles later on. So X and Y, we're going to use this variable.
9:01:16
How would you say nomenclature as capital X and Y. Why is that? Because X is typically a matrix features and labels. So let's go here. Mate circles. And we're going to make N samples. So 1000 different samples. We're going to add some noise in there. Just put a little bit of randomness. Why not? You can increase this as you want. I found that 0.03 is fairly good for what we're doing. And then we're going to also pass in the random state variable, which is equivalent to sitting a random
9:01:46
or setting a random seed. So we're flavoring the randomness here. Wonderful. So now let's have a look at the length of X, which should be what? And length of Y. Oh, we don't have Y underscore getting a bit trigger happy with this keyboard here. 1000. So we have 1000 samples of X caught with 1000 or paired with 1000 samples of Y features labels. So let's have a look at the first five of X. So print first five samples of X. And then we'll put in here X. And we can index
9:02:24
on this five because we're adhering to the data, explorer's motto of visualize visualize visualize
9:02:34
first five samples of Y. And then we're going to go why same thing here. Wonderful. Let's have a look. Maybe we'll get a new line in here. Just so
9:02:50
looks a bit better. Wonderful. So numerical. Our samples are already numerical. This is one of the reasons why we're creating our own data set. We'll see later on how we get non numerical data into numbers. But for now, our data is numerical, which means we can learn it with our model or we can build a model to learn patterns in here. So this sample has the label of one. And this sample has the label of one as well. Now, how many features do we have per sample? If I highlight
9:03:19
this line, how many features is this? It would make it a bit easier if there was a comma here, but we have two features of X, which relates to one label of Y. And so far, we've only seen, let's have a look at all of Y. We've got zero on one. So we've got two classes. What does this mean? Zero or one? One thing or another? Well, it looks like binary classification to me, because we've got only zero or only one. If there was zero, one, two, it would be
9:03:53
multi class classification, because we have more than two things. So let's X out of this. Let's keep going and do a little bit more data exploration. So how about we make a data frame? With pandas of circle data. There is truly no real definite way of how to explore data. For me, I like to visualize it multiple different ways, or even look at random samples. In the case of large data sets, such as images or text or whatnot. If you have 10 million samples, perhaps
9:04:26
visualizing them one by one is not the best way to do so. So random can help you out there. So we're going to create a data frame, and we can insert a dictionary here. So I'm going to call the features in this part of X, X1, and these are going to be X2. So let's say I'll write some code to index on this. So everything in the zero index will be X1. And everything in the first index, there we go, will be X2. Let me clean up this code. This should be on different lines,
9:05:05
enter. And then we've got, let's put in the label as Y. So this is just a dictionary here. So X1 key to X0. X2, a little bit confusing because of zero indexing, but X feature one, X feature two, and the label is Y. Let's see what this looks like. We'll look at the first 10 samples.
9:05:29
Okay, beautiful. So we've got X1, some numerical value, X2, another numerical value, correlates to or matches up with label zero. But then this one, 0442208, and negative that number matches up with label zero. So I can't tell what the patterns are just looking at these numbers. You might be able to, but I definitely can't. We've got some ones. All these numbers look the same to me. So what can we do next? Well, how about we visualize, visualize, visualize, and instead of just numbers
9:06:04
in a table, let's get graphical this time, visualize, visualize, visualize. So we're going to bring in our friendly mapplotlib, import mapplotlib, which is a very powerful plotting library. I'm just going to add some cells here. So we've got some space, mapplotlib.pyplot as PLT. That's right. We've got this plot.scatter. We're going to do a scatterplot equals X. And we want the first index. And then Y is going to be X as well. So that's going to appear on the Y axis. And then we want to
9:06:43
color it with labels. We're going to see what this looks like in a second. And then the color map,
9:06:49
C map stands for color map is going to be plot dot color map PLT. And then red, yellow, blue, one of my favorite color outputs. So let's see what this looks like. You ready?
9:07:03
Ah, there we go. There's our circles. That's a lot better for me. So what do you think we're going to try and do here? If this is our data and we're working on classification,
9:07:16
we're trying to predict if something is one thing or another. So our problem is we want to try and separate these two circles. So say given a number here or given two numbers and X one and an X two, which are coordinates here, we want to predict the label. Is it going to be a blue dot or is it going to be a red dot? So we're working with binary classification. So we have one thing or another. Do we have a blue dot or a red dot? So this is going to be our toy data here.
9:07:46
And a toy problem is, let me just write this down. This is a common thing that you'll also hear in machine learning. Note, the data we're working with is often referred to as a toy data set,
9:08:02
a data set that is small enough to experiment on, but still sizable enough to practice the fundamentals. And that's what we're really after in this notebook is to practice the fundamentals of neural network classification. So we've got a perfect data set to do this. And by the way, we've got this from scikit-learn. So this little function here made all of these samples for us. And how could you find out more about this function here? Well, you could go scikit-learn
9:08:39
classification data sets. There are actually a few more in here that we could have done. I just like the circle one. Toy data sets, we saw that. So this is like a toy box of different data sets. So if you'd like to learn more about some data sets that you can have a look in here and potentially practice on with neural networks or other forms of machine learning models from scikit-learn, check out this scikit-learn. I can't speak highly enough. I know this is a pie-torch
9:09:04
course. We're not focused on this, but they kind of all come together in terms of the machine learning and deep learning world. You might use something from scikit-learn, like we've done here, to practice something. And then you might use pie-torch for something else, like what we're doing here. Now, with that being said, what are the input and output shapes of our problem?
9:09:25
Have a think about that. And also have a think about how we'd split this into training and test.
9:09:31
So give those a go. We covered those concepts in some previous videos, but we'll do them together in the next video. I'll see you there. Welcome back. In the last video, we made some classification data so that we can practice building a neural network in pie-torch to separate the blue dots from the red dots. So let's keep pushing forward on that. And I'll just clean up here a little bit, but where are we in our workflow? What have we done so far? Well, we've got our data ready a
9:10:02
little bit. We haven't turned it into tenses. So let's do that in this video, and then we'll keep pushing through all of these. So in here, I'm going to make this heading 1.1. Check input and output shapes. The reason we're focused a lot on input and output shapes is why, because machine learning deals a lot with numerical representations as tenses. And input and output shapes are some of the most common errors, like if you have a mismatch between your input and
9:10:32
output shapes of a certain layer of an output layer, you're going to run into a lot of errors there. So that's why it's good to get acquainted with whatever data you're using, what are the input shapes and what are the output shapes you'd like. So in our case, we can go x dot shape and y dot shape. So we're working with NumPy arrays here if we just look at x. That's what the make circles function is created for us. We've got an array, but as our workflow says,
9:11:02
we'd like it in tenses. If we're working with PyTorch, we want our data to be represented as PyTorch tenses of that data type. And so we've got a shape here, we've got a thousand samples, and x has two features, and y has no features. It's just a single number. It's a scalar. So it doesn't have a shape here. So there's a thousand samples of y, thousand samples of x, two samples of x equals one y label. Now, if you're working with a larger problem, you might have a thousand
9:11:30
samples of x, but x is represented by 128 different numbers, or 200 numbers, or as high as you want, or just 10 or something like that. So just keep in mind that this number is quite flexible of how many features represent a label. Why is the label here? But let's keep going. So view the first example of features and labels. So let's make it explicit with what we've just been discussing. We'll write some code to do so. We'll get the first sample of x, which is the zero index,
9:12:06
and we'll get the first sample of y, which is also the zero index. We could get really anyone because they're all of the same shape. But print values for one sample of x. What does this equal? X sample, and the same for y, which is y sample. And then we want to go print f string for one sample of x. We'll get the shape here. X sample dot shape, and the same for y, and then we'll get y sample dot shape. Beautiful. What's this going to do? Well, we've got one sample of x. So this
9:12:53
sample here of these numbers, we've got a lot going on here. 75424625 and 0231 48074. I mean, you can try to find some patterns in those. If you do, all the best here, and the same for y. So this is, we have the y sample, this correlates to a number one, a label of one. And then we have shapes for one sample of x, which is two. So we have two features for y. It's a little bit confusing here because y is a scalar, which doesn't actually have a shape. It's just one value. So for me,
9:13:26
in terms of speaking this, teaching it out loud, we'll be two features of x trying to predict one number for y. And so let's now create another heading, which is 1.2. Let's get our data into tenses, turn data into tenses. We have to convert them from NumPy. And we also want to create train and test splits. Now, even though we're working with a toy data set here, the principle of turning data into tenses and creating train and test splits will stay around for almost any
9:13:57
data set that you're working with. So let's see how we can do that. So we want to turn data into tenses. And for this, we need to import torch, get pytorch and we'll check the torch version. It has to be at least 1.10. And I might just put this down in the next cell. Just make sure we can import pytorch. There we go, 1.10 plus CUDA 111. If your version is higher than that, that is okay. The code below should still work. And if it doesn't, let me know. So x equals torch dot
9:14:34
from NumPy. Why are we doing this? Well, it's because x is a NumPy array. And if we go x dot, does it have a d type attribute float 64? Can we just go type or maybe type? Oh, there we go. NumPy and DRA. We can just go type x. NumPy and DRA. So we want it in a torch tensor. So we're going to go from NumPy. We saw this in the fundamental section. And then we're going to change it into type torch dot float. A float is an alias for float 32. We could type the same thing. These two are
9:15:12
equivalent. I just going to type torch float for writing less code. And then we're going to go the same with why torch from NumPy. Now, why do we turn it into a torch float? Well, that's because if you recall, the default type of NumPy arrays is, if we go might just put out this in a comma x dot D type is float 64. There we go. However, pytorch, the default type is float 32. So we're changing it into pytorch's default type. Otherwise, if we didn't have this little
9:15:46
section of code here dot type torch dot float, our tensors would be of float 64 as well. And that may cause errors later on. So we're just going for the default data type within pytorch. And so now let's have a look at the first five values of x and the first five values of y. What do we have? Beautiful. We have tensor data types here. And now if we check the data type of x and we check the data type of y, what do we have? And then one more, we'll just go type x. So we have
9:16:19
our data into tensors. Wonderful. But now so it's torch dot tensor. Beautiful. But now we would like training and test sets. So let's go split data into training and test sets. And a very, very popular way to split data is a random split. So before I issued the challenge of how you would split this into a training and test set. So because these data points are kind of scattered all over the place, we could split them randomly. So let's see what that looks like. To do so, I'm going to
9:16:58
use our faithful scikit learn again. Remember how I said scikit learn has a lot of beautiful methods and functions for a whole bunch of different machine learning purposes. Well, one of them is for a train test split. Oh my goodness, pytorch I didn't want auto correct there. Train test split. Now you might be able to guess what this does. These videos are going to be a battle between me and code labs auto correct. Sometimes it's good. Other times it's not. So we're going to set this code
9:17:26
up. I'm going to write it or we're going to write it together. So we've got x train for our training features and X tests for our testing features. And then we also want our training labels and our testing labels. That order is the order that train test split works in. And then we have train test split. Now if we wrote this function and we wanted to find out more, I can press command ship space, which is what I just did to have this. But truly, I don't have a great time reading all
9:17:53
of this. You might. But for me, I just like going train test split. And possibly one of the first functions that appears, yes, is scikit learn. How good is that? So scikit learn dot model selection dot train test split. Now split arrays or matrices into random train and test subsets. Beautiful. We've got a code example of what's going on here. You can read what the different parameters do. But we're going to see them in action. This is just another example of where machine learning
9:18:23
libraries such as scikit learn, we've used matplotlib, we've used pandas, they all interact together to serve a great purpose. But now let's pass in our features and our labels. This is the order that they come in, by the way. Oh, and we have the returns splitting. So the order here, I've got the order goes x train x test y train y test took me a little while to remember this order. But once you've created enough training test splits with this function,
9:18:51
you kind of know this off by heart. So just remember features first train first and then labels.
9:18:57
And we jump back in here. So I'm going to put in the test size parameter of 0.2. This is percentage wise. So let me just write here 0.2 equals 20% of data will be test. And 80% will be train. If we wanted to do a 50 50 split, that kind of split doesn't usually happen, but you could go 0.5. But the test size says, hey, how big and percentage wise do you want your test data to be? And so behind the scenes train test split will calculate what's 20% of
9:19:33
our x and y samples. So we'll see how many there is in a second. But let's also put a random state in here. Because if you recall back in the documentation, train test split splits data randomly into random train and test subsets. And random state, what does that do for us? Well, this is a random seed equivalent of very similar to torch dot manual seed. However, because we are using scikit learn, setting torch dot manual seed will only affect pytorch code rather than
9:20:07
scikit learn code. So we do this so that we get similar random splits. As in, I get a similar random split to what your random split is. And in fact, they should be exactly the same. So let's run this. And then we'll check the length of x train. And length of x test. So if we have 1000 total samples, and I know that because above in our make circles function, we said we want 1000 samples, that could be 10,000, that could be 100. That's the beauty of creating your own
9:20:40
data set. And we have length y train. If we have 20% testing values, how many samples are going to be dedicated to the test sample, 20% of 1000 years, 200, and 80%, which is because training is going to be training here. So 100 minus 20% is 80%. So 80% of 1000 years, let's find out.
9:21:07
Run all beautiful. So we have 800 training samples, 200 testing samples. This is going to be the data set that we're going to be working with. So in the next video, we've now got training and test sets, we've started to move through our beautiful pytorch workflow here. We've got our data ready, we've turned it into tenses, we've created a training and test split. Now it's time to build or pick a model. So I think we're still in the building phase. Let's do that in the next
9:21:36
video. Welcome back. In the last video, we split our data into training and test sets. And because we did 80 20 split, we've got about 800 samples to train on, and 200 samples to test on. Remember, the training set is so that the model can learn patterns, patterns that represent this data set here, the circles data set, red dots or blue dots. And the test data set is so that we can evaluate those patterns. And I took a little break before, but you can tell that because my
9:22:10
notebook is disconnected. But if I wanted to reconnect it, what could I do? We can go here, run time, run before that's going to run all of the cells before. It shouldn't take too long because we haven't done any large computations. But this is good timing because we're up to part two, building a model. And so there's a fair few steps here, but nothing that we haven't covered before, we're going to break it down. So let's build a model to classify our blue and red dots.
9:22:43
And to do so, we want to tenses. I want to not tenses. That's all right. So let me just make some space here. There we go. So number one, let's set up device agnostic code. So we get in the habit of creating that. So our code will run on an accelerator. I can't even spell accelerator. It doesn't matter. You know what I mean? GPU. If there is one. Two. What should we do next? Well, we should construct a model. Because if we want to build a model, we need a model.
9:23:24
Construct a model. And we're going to go by subclassing and then dot module. Now we saw this in the previous section, we subclassed and then module. In fact, all models in PyTorch subclass and end up module. And let's go define loss function and optimizer.
9:23:47
And finally, good collabs auto correct is not ideal. And then we'll create a training and test loop. Though this will probably be in the next section. We'll focus on building a model here. And of course, all of these steps are in line with what they're in line with this. So we don't have device agnostic code here, but we're just going to do it enough so that we have a habit. These are the main steps. Pick or build a pre-trained model, suit your problem, pick a
9:24:13
loss function and optimizer, build a training loop. So let's have a look. How can we start this off? So we will import PyTorch. And and then we've already done this, but we're going to do it anyway for completeness, just in case you wanted to run your code from here, import and then. And we're going to make device agnostic code. So we'll set the device equal to CUDA if torch dot CUDA
9:24:45
is available else CPU, which will be the default. The CPU is the default. If there's no GPU, which means that CUDA is available, all of our PyTorch code will default to using the CPU device. Now we haven't set up a GPU yet so far. You may have, but as you see, my target device is currently CPU. How about we set up a GPU? We can go into here runtime change runtime type GPU. And I'm going to click save. Now this is going to restart the runtime and reconnect.
9:25:18
So once it reconnects beautiful, we could actually just run this code cell here. This is going to set up the GPU device, but because we're only running this cell, if we were to just set up X train, we've not been defined. So because we restarted our runtime, let's run all or we can just run before. So this is going to rerun all of these cells here. And do we have X train now? Let's have a look. Wonderful. Yes, we do. Okay, beautiful. So we've got device agnostic code.
9:25:51
In the next video, let's get on to constructing a model. I'll see you there.
9:25:58
Welcome back. In the last video, we set up some device agnostic code. So this is going to come in later on when we send our model to the target device, and also our data to the target device. This is an important step because that way, if someone else was able to run your code or you were to run your code in the future, because we've set it up to be device agnostic, quite a fault it will run on the CPU. But if there's an accelerator present,
9:26:22
well, that means that it might go faster because it's using a GPU rather than just using a CPU. So we're up to step two here construct a model by subclassing and in module. I think we're going to write a little bit of text here just to plan out the steps that we're doing. Now we've set up device agnostic code. Let's create a model that we're going to break it down. We've got some sub steps up here. We're going to break it down even this one down into some sub-sub steps. So number
9:26:55
one is we're going to subclass and then got module. And a reminder here, I want to make some space, just so we're coding in about the middle of the page. So almost all models in pytorch, subclass, and then got module because there's some great things that it does for us behind the scenes. And step two is we're going to create two and then dot linear layers. And we want these that are capable to handle our data. So that are capable of handling the shapes of our data.
9:27:37
Step three, we want to define or defines a forward method. Why do we want to define a forward method? Well, because we're subclassing an end dot module, right? And so the forward method defines a forward method that outlines the forward pass or forward computation of the model. And number four, we want to instantiate, well, this doesn't really have to be the part of creating it, but we're going to do anyway, and instantiate an instance of our model class and send it to the target device. So I'm
9:28:27
going to be a couple of little different steps here, but nothing too dramatic that we haven't really covered before. So let's go number one, construct a model that subclasses an end dot module. So I'm going to code this all out. Well, we're going to code this all out together. And then we'll go back through and discuss it, and then maybe draw a few pictures or something to check out what's actually happening. So circle model V one, because we're going to try and split some circles,
9:28:56
red and blue circles. This is our data up here. This is why it's called circle model, because we're trying to separate the blue and red circle using a neural network. So we've subclassed an end dot module. And when we create a class in Python, we'll create a constructor here, a net, and then put in super dot underscore a net. And then inside the constructor, we're going to create our layers. So this is number two, create two and then linear layers, capable of handling the shapes
9:29:27
of our data. So I'm going to write this down here to create two, two, and then dot linear layers, capable of handling the shapes of our data. And so if we have a look at X train, what are the shapes here? What's the input shape? Because X train is our features, right? Now features are going to go into our model. So we have 800 training samples. This is the first number here of size two each. So 800 of these and inside each is two numbers. Again, depending on the data set you're working
9:30:07
with, your features may be 100 in length, a vector of 100, or maybe a different size tensor all together, or there may be millions. It really depends on what data set you're working with. Because we're working with a simple data set, we're going to focus on that. But the principal is still the same. You need to define a neural network layer that is capable of handling your input features. So we're going to make layer one equals an n dot linear. And then if we wanted
9:30:33
to find out what's going on an n n dot linear, we could run shift command space on my computer, because it's a Mac, maybe shift control space if you're on Windows. So we're going to define the n features. What would n features be here? Well, we just decided that our X has two features. So n features are going to be two. And now what is the out features? This one is a little bit tricky.
9:30:58
So in our case, we could have out features equal to one if we wanted to just pass a single linear layer, but we want to create two linear layers here. So why would out features be one? Well, that's because if we have a look at the first sample of Y train, we would want us to input, or maybe we'll look at the first five. We want to map one sample of X to one sample of Y and Y has a shape of one. Oh, well, really, it's nothing because it's a scalar, but we would still put
9:31:29
one here so that it outputs just one number. But we're going to change this up. We're going to put it into five and we're going to create a second layer. Now, this is an important point of joining together neural networks in features here. What do you think the in features of our second layer is going to be? If we've produced an out feature of five here, now this number is arbitrary. We could do 128. We could do 256. Generally, it's multiples of 8, 64. We're just doing five now because we're
9:32:00
keeping it nice and simple. We could do eight multiples of eight is because of the efficiency of computing. I don't know enough about computer hardware to know exactly why that's the case, but that's just a rule of thumb in machine learning. So the in features here has to match up with the out features of a previous layer. Otherwise, we'll get shape mismatch errors. And so let's go here out features. So we're going to treat this as the output layer. So this is the out features equals
9:32:26
one. So takes in two features and upscales to five features. So five numbers. So what this does, what this layer is going to do is take in these two numbers of X, perform an end up linear. Let's have a look at what equation it does. An end up linear is going to perform this function here on the inputs. And it's going to upscale it to five features. Now, why would we do that? Well, the rule of thumb here, because this is denoted as a hidden unit, or how many hidden neurons there
9:33:01
are. The rule of thumb is that the more hidden features there are, the more opportunity our model has to learn patterns in the data. So to begin with, it only has two numbers to learn patterns on, but at when we upscale it to five, it has five numbers to learn patterns on. Now, you might think, why don't we just go straight to like 10,000 or something? But there is like an upper limit here to sort of where the benefits start to trail off. We're just using five because it keeps it nice
9:33:27
and simple. And then the in features of the next layer is five, so that these two line up. We're going to map this out visually in a moment, but let's keep coding. We've got in features two for X. And now this is the output layer. So takes in five features from previous layer and outputs a single feature. And now this is same shape. Same shape as why. So what is our next step? We want to define a Ford method, a Ford computation of Ford pass. So the Ford method is going to
9:34:01
define the Ford computation. And as an input, it's going to take X, which is some form of data. And now here's where we can use layer one and layer two. So now let's just go return.
9:34:14
Or we'll put a note here of what we're doing. Three, we're going to go define a Ford method that outlines the Ford pass. So Ford, and we're going to return. And here's some notation we haven't quite seen yet. And then we're going to go self layer two. And inside the brackets we'll have self layer one inside those brackets. We're going to have X. So the way this goes is X goes into layer one. And then the output of layer one goes into layer two. So whatever data we have,
9:34:50
so our training data, X train goes into layer one performs the linear calculation here. And then it goes into layer two. And then layer two is going to output, go to the output. So X is the input, layer one computation layer two output. So we've done that. Now let's do step four, which is instantiate an instance of our model class. And send it to the target device. So this is our model class, circle model V zero. We're just going to create a model because it's the first model we've
9:35:33
created up this section. Let's call it model zero. And we're going to go circle model V one. And then we're going to go to two. And we're going to pass in device, because that's our target device. Let's now have a look at model zero. And then Oh, typo. Yeah, classic. What did we get wrong here? Oh, did we not pass in self self? Oh, there we go.
9:36:06
Little typo classic. But the beautiful thing about creating a class here is that we could put this into a Python file, such as model dot pi. And then we wouldn't necessarily have to rewrite this all the time, we could just call it. And so let's just check what the vice it's on. So target device is CUDA, because we've got a GPU, thank you, Google Colab. And then if we wanted to, let's go next model zero dot parameters, we'll call the parameters, and then we'll go device.
9:36:38
CUDA beautiful. So that means our models parameters are on the CUDA device. Now we've covered enough code in here for this video. So if you want to understand it a little bit more, go back through it. But we're going to come back in the next video and make it a little bit more visual. So I'll see you there. Welcome back. In the last video, we did something very, very exciting. We created our first multi layer neural network. But right now, this is just code on a page. But truly, this is
9:37:07
what the majority of building machine learning models in PyTorch is going to look like. You're going to create some layers, or a simple or as complex as you like. And then you're going to use those layers in some form of Ford computation to create the forward pass. So let's make this a little bit more visual. If we go over to the TensorFlow playground, and now TensorFlow is another deep learning framework similar to PyTorch, it just allows you to write code such as this,
9:37:38
to build neural networks, fit them to some sort of data to find patterns and data, and then use those machine learning models in your applications. But let's create this. Oh, by the way, this is playground.tensorFlow.org. This is a neural network that we can train in the browser if we really wanted to. So that's pretty darn cool. But we've got a data set here, which is kind of similar to the data set that we're working with. We have a look at our circles one.
9:38:07
Let's just say it's close enough. It's circular. That's what we're after. But if we increase this, we've got five neurons now. We've got two features here, X1 and X2. Where is this reminding you of what's happening? There's a lot of things going on here that we haven't covered yet, but don't worry too much. We're just focused on this neural network here. So we've got some features as the input. We've got five hidden units. This is exactly what's going on with the model
9:38:31
that we just built. We pass in X1 and X2, our values. So if we go back to our data set, these are X1 and X2. We pass those in. So we've got two input features. And then we pass them to a hidden layer, a single hidden layer, with five neurons. What have we just built? If we come down into here to our model, we've got in features two, out features five. And then that feeds into another layer, which has in features five and out features one. So this is the exact same model
9:39:05
that we've built here. Now, if we just turn this back to linear activation, because we're sticking with linear for now, we'll have a look at different forms of activation functions later on. And maybe we put the learning rate, we've seen the learning rate to 0.01. We've got epochs here, got classification. And we're going to try and fit this neural network to this data. Let's see what happens. Oh, the test loss, it's sitting about halfway 0.5. So about 50% loss. So if we only have two
9:39:37
classes and we've got a loss of 50%, what does that mean? Well, the perfect loss was zero. And the worst loss was one. Then we just divide one by two and get 50%. But we've only got two classes. So that means if our model was just randomly guessing, it would get a loss of about 0.5, because you could just randomly guess whatever data point belongs to blue or orange in this case. So in a binary classification problem, if you have the same number of samples in each class,
9:40:07
in this case, blue dots and orange dots, randomly guessing will get you about 50%. Just like tossing a coin, toss a coin 100 times and you get about 50 50 might be a little bit different, but it's around about that over the long term. So we've just fit for 3000 epochs. And we're still not getting any better loss. Hmm. I wonder if that's going to be the case for our neural network. And so to draw this in a different way, I'm going to come to a little tool called fig jam, which is just a
9:40:35
whiteboard that we can put shapes on and it's based on the browser. So this is going to be nothing fancy. It's going to be a simple diagram. Say this is our input. And I'm going to make this green because my favorite color is green. And then we're going to have, let's make some different colored dots. I want a blue dot here. So this can be dot one, and dot two, I'll put another dot here. I'll zoom out a little so we have a bit more space. Well, maybe that was too much. 50%
9:41:11
looks all right. So let me just move this around, move these up a little. So we're building a neural network here. This is exactly what we just built. And so we'll go here. Well, maybe we'll put this as input X one. So this will make a little bit more sense. And then we'll maybe we can copy this. Now this is X two. And then we have some form of output. Let's make this one. And we're going to color this orange. So output. Right. So you can imagine how we got connected dots here.
9:41:48
They will connect these. So our inputs are going to go through all of these. I wonder if I can draw here. Okay, this is going to be a little bit more complex, but that's all right. So this is what we've done. We've got two input features here. And if we wanted to keep drawing these, we could all of these input features are going to go through all of these hidden units that we have. I just drew the same arrow twice. That's okay. But this is what's happening in the forward
9:42:14
computation method. It can be a little bit confusing for when we coded it out. Why is that? Well, from here, it looks like we've only got an input layer into a single hidden layer in the blue. And an output layer. But truly, this is the same exact shape. You get the point. And then all of these go to the output. But we're going to see this computationally later on. So whatever data set you're working with, you're going to have to manufacture some form of input layer. Now this
9:42:45
may be you might have 10 of these if you have 10 features. Or four of them if you have four features. And then if you wanted to adjust these, well, you could increase the number of hidden units or the number of out features of a layer. What just has to match up is that the layer it's going into has to have a similar shape as the what's coming out of here. So just keep that in mind as you're going on. And in our case, we only have one output. So we have the output here,
9:43:15
which is why. So this is a visual version. We've got the TensorFlow playground. You could play around with that. You can change this to increase. Maybe you want five hidden layers with five neurons in each. This is a fun way to explore. This is a challenge, actually, go to playground.tensorflow.org, replicate this network and see if it fits on this type of data. What do you think, will it? Well, we're going to have to find out in the next few videos. So I'm going to show you in the
9:43:43
next video another way to create the network that we just created. This one here with even less code than what we've done before. I'll see you there. Welcome back. In the last video, what we discussed, well, actually, in the previous video to last, we coded up this neural network here, circle model V zero. By subclassing an end or module, we created two linear layers, which are capable of handling the shape of our data in features two because why we have two X features.
9:44:14
Out features were upscaling the two features to five so that it gives our network more of a chance to learn. And then because we've upscaled it to five features, the next subsequent layer has to be able to handle five features as input. And then we have one output feature because that's the same shape as our Y here. Then we got a little bit visual by using the TensorFlow playground. Did you try out that challenge, make five in layers with five neurons? Did it work?
9:44:41
And then we also got a little bit visual in Figma as well. This is just another way of visualizing different things. You might have to do this a fair few times when you first start with neural networks. But once you get a bit of practice, you can start to infer what's going on through just pure code. So now let's keep pushing forward. How about we replicate this with a simpler way? Because our network is quite simple, that means it only has two layers.
9:45:07
That means we can use. Let's replicate the model above using nn.sequential. And I'm going to code this out. And then we can look up what nn.sequential is. But I think you'll be able to comprehend what's happening by just looking at it. So nn, which is torch.nn. We can do torch.nn, but we've already imported nn. We're going to call nn.sequential. And then we're going to go nn.linear. And what was the in features of our nn.linear? Well, it was two because we have two in features. And then
9:45:45
we're going to replicate the same out features. Remember, we could customize this to whatever we want 10, 100, 128. I'm going to keep it at five, nice and simple. And then we go nn.linear. And the in features of this next layer is going to be five because the out features of the previous layer was five. And then finally, the out features here is going to be one because we want one y value to our two x features. And then I'm going to send that to the device. And then I'm going to
9:46:15
have a look at model zero. So this is, of course, going to override our previous model zero. But have a look. The only thing different is that this is from the circle model V zero class. We subclassed an n dot module. And the only difference is the name here. This is just from sequential. And so can you see what's going on here? So as you might have guessed sequential, it implements most of this code for us behind the scenes. Because we've told it that it's going
9:46:49
to be sequential, it's just going to go, hey, step the code through this layer, and then step the code through this layer. And outputs basically the same model, rather than us creating our own forward method, you might be thinking, Daniel, why don't you show us this earlier? That looks like such an easy way to create a neural network compared to this. Well, yes, you're 100% right. That is an easier way to create a neural network. However, the benefit of subclassing, and that's
9:47:17
why I started from here, is that when you have more complex operations, such as things you'd like to construct in here, and a more complex forward pass, it's important to know how to build your own subclasses of nn dot module. But for simple straightforward stepping through each layer one by one, so this layer first, and then this layer, you can use nn dot sequential. In fact, we could move this code up into here. So we could do this self dot, we'll call this
9:47:49
two linear layers equals nn dot sequential. And we could have layer one, we could go self,
9:48:01
self dot layer one. And or actually, we'll just recode it, we'll go nn dot linear. So it's so it's the same code is what we've got below in features. If I could type that'll be great, n features equals two, out features equals five. And then we go nn dot linear. And then we go n features equals what equals five, because it has to line up out features equals one.
9:48:32
And then we've got two linear layers. And then if we wanted to get rid of this, return to linear layers, and we'll pass it X remake it. There we go. Well, because we've created these as well, let's get rid of that. Beautiful. So that's the exact same model, but just using nn dot sequential. Now I'm going to get rid of this so that our code is not too verbose. That means a lot going on. But this is the flexibility of PyTorch. So just keep in mind that there's a fair few ways
9:49:08
to make a model. The simplest is probably sequential. And then subclassing is this is a little bit more complicated than what we've got. But this can extend to handle lot more complex neural networks, which you'll likely have to be building later on. So let's keep pushing forward. Let's see what happens if we pass some data through here. So we'll just rerun this cell to make sure we've got our model zero instantiated. We'll make some predictions with the model. So of course, if we
9:49:38
have a look at our model zero dot state deck, oh, this will be a good experiment. So look at this. So we have weight, a weight tensor, a bias tensor, a weight tensor, and a bias tensor. So this is for the first of the zeroth layer, these two here with the zero dot, and then the one dot weight is four, of course, the first index layer. Now have a look inside here. Now you see how out features is five. Well, that's why our bias parameter has five values here. And the same thing for this weight
9:50:14
value here. And the weight value here, why is this have 10 samples? One, two, three, four, five, six, seven, eight, nine, 10, because two times five equals 10. So this is just with a simple two layer network, look at all the numbers that are going on behind the scenes. Imagine coding all of these by hand. Like there's something like 20 numbers or something here. Now we've only done two layers here. Now the beauty of this is that in the previous section, we created all of the weight
9:50:48
and biases using an end dot parameter and random values. You'll notice that these are all random two. Again, if yours are different to mine, don't worry too much, because they're going to be started randomly and we haven't set a random seed. But the thing to note here is that PyTorch is creating all of these parameters for us behind the scenes. And now when we do back propagation and gradient descent, when we code our training loop, the optimizer is going to change all of these values
9:51:15
ever so slightly to try and better fit or better represent the data so that we can split our two circles. And so you can imagine how verbose this could get if we had say 50 layers with 128 different features of each. So let's change this up, see what happens. Watch how quickly the numbers get out of hand. Look at that. We just changed one value and look how many parameters our model has. So you might be able to calculate that by hand, but I personally don't want to. So we're going to
9:51:48
let PyTorch take care of a lot of that for us behind the scenes. So for now we're keeping it simple, but that's how we can crack our models open and have a look at what's going on. Now that was a little detour. It's time to make some predictions with random numbers. I just wanted to highlight the fact that our model is in fact instantiated with random numbers here. So the untrained threads model zero, we're going to pass in X test. And of course, we have to send the test data to the
9:52:15
device. Otherwise, if it's on a different device, we'll get errors because PyTorch likes to make calculations on the same device. So we'll go print. Let's do a nice print statement of length of predictions. We're going to go length or then untrained threads, we'll pass that in there. And then we'll go, oh no, we need to squiggle. And then we'll go shape. Shape is going to be untrained spreads dot shape. So this is again, following the data explorer's motto of visualize,
9:52:51
visualize, visualize. And sometimes print is one of the best ones to do so. So length of test samples,
9:52:58
you might already know this, or we've already checked this together, haven't we? X test.
9:53:04
And then we're going to get the shape, which is going to be X test dot shape. Wonderful. And then we're going to print. What's our little error here? Oh no, collabs tricking me. So let's go first 10 predictions. And we're going to go untrained threads. So how do you think these predictions will fare? They're doing it with random numbers. And what are we trying to predict again? Well, we're trying to predict whether a dot is a red dot or a blue dot or zero or one. And then we'll go
9:53:41
first 10 labels is going to be, we'll get this on the next line. And we'll go Y test.
9:53:52
Beautiful. So let's have a look at this untrained predictions. So we have length of predictions is 200. Length of test samples is 200. But the shapes are different. What's going on here? Y test. And let's have a look at X test. Oh, well, I better just have a look at Y test.
9:54:18
Why don't we have a two there? Oh, I've done X test dot shape. Oh, let's test samples. That's okay. And then the predictions are one. Oh, yes. So Y test. Let's just check the first 10 X test. So a little bit of clarification needed here with your shapes. So maybe we'll get this over here because I like to do features first and then labels. What did we miss here? Oh, X test 10 and Y test. See, we're troubleshooting on the fly here. This is what you're going to do with
9:54:50
a lot of your code. So there's our test values. There's the ideal labels. But our predictions, they don't look like our labels. What's going on here? We can see that they're on the CUDA device, which is good. We said that. We can see that they got gradient tracking. Oh, we didn't with touch. We didn't do inference mode here. That's a poor habit of us. Excuse me. Let's inference mode this. There we go. So you notice that the gradient tracking goes away there. And so our
9:55:16
predictions are nowhere near what our test labels are. But also, they're not even in the same like ball park. Like these are whole numbers, one or zero. And these are all floats between one and zero. Hmm. So maybe rounding them. Will that do something? So where's our threads here? So we go torch dot round. What happens there? Oh, they're all zero. Well, our model is probably going to get about 50% accuracy. Why is that? Because all the predictions look like they're
9:55:49
going to be zero. And they've only got two options, basically head or tails. So when we create our model and when we evaluate it, we want our predictions to be in the same format as our labels. But we're going to cover some steps that we can take to do that in a second. What's important to take away from this is that there's another way to replicate the model we've made above using nn dot sequential. We've just replicated the same model as what we've got here. And n dot
9:56:15
sequential is a simpler way of creating a pytorch model. But it's limited because it literally just sequentially steps through each layer in order. Whereas in here, you can get as creative as you want with the forward computation. And then inside our model, pytorch has behind the scenes created us some weight and bias tensors for each of our layers with regards to the shapes that we've set. And so the handy thing about this is that if we got quite ridiculous with our layers,
9:56:46
pytorch would still do the same thing behind the scenes, create a whole bunch of random numbers for us. And because our numbers are random, it looks like our model isn't making very good predictions. But we're going to fix this in the next few videos when we move on to
9:57:02
fitting the model to the data and making a prediction. But before we do that, we need to pick up a loss function and an optimizer and build a training loop. So let's get on to these two things.
9:57:13
Welcome back. So over the past few videos, we've been setting up a classification model to deal with our specific shape of data. Now recall, depending on the data set that you're working with will depend on what layers you use for now we're keeping it simple and n dot linear is one of the most simple layers in pytorch. We've got two input features, we're upscaling that to five output features. So we have five hidden units, and then we have one output feature. And that's in line
9:57:38
with the shape of our data. So two features of x equals one number for y. So now let's continue on modeling with where we're up to. We have build or pick a model. So we've built a model. Now we need to pick a loss function and optimizer. We're getting good at this. So let's go here, set up loss function and optimizer. Now here comes the question. If we're working on classification previously, we used, let's go to the next one, and an dot L one loss for regression, which is
9:58:13
MAE mean absolute error, just a heads up that won't necessarily work with a classification problem. So which loss function or optimizer should you use? So again, this is problem specific. But with a little bit of practice, you'll get used to using different ones. So for example, for regression,
9:58:38
you might want, which is regressions predicting a number. And I know it can get fusing because it looks like we're predicting a number here, we are essentially predicting a number. But this relates to a class. So for regression, you might want MAE or MSE, which is mean absolute
9:58:56
absolute error, or mean squared error. And for classification, you might want binary cross entropy
9:59:09
or categorical cross entropy, which is sometimes just referred to as cross entropy. Now, where would you find these things out? Well, through the internet, of course. So you could go, what is binary cross entropy? I'm going to leave you this for your extra curriculum to read through this. We've got a fair few resources here. Understanding binary cross entropy slash log loss
9:59:36
by Daniel Godoy. Oh, yes. Great first name, my friend. This is actually the article that I would recommend to if you want to learn what's going on behind the scenes through binary cross entropy. For now, there's a lot of math there. We're going to be writing code to implement this. So PyTorch has done this for us. Essentially, what does a loss function do? Let's remind ourselves.
9:59:58
Go down here. As a reminder, the loss function measures how wrong your models' predictions are. So I also going to leave a reference here to I've got a little table here in the book version of this course. So 0.2 neural network classification with PyTorch set up loss function and optimizer. So we've got some example loss functions and optimizers here, such as stochastic gradient descent or SGD optimizer, atom optimizer is also very popular. So I've got problem type here,
10:00:32
and then the PyTorch code that we can implement this with. We've got binary cross entropy loss. We've got cross entropy loss, mean absolute error, MAE, mean squared error, MSE. So you want to use these two for regression. There are other different loss functions you could use, but these are some of the most common. That's what I'm focusing on, the most common ones that are going to get you through a fair few problems. We've got binary classification, multi-class classification. What
10:00:57
are we working with? We're working with binary classification. So we're going to look at torch.nn BCE, which stands for binary cross entropy, loss with logits. What the hell is a logit? And BCE loss. Now this is confusing. Then trust me, when I first started using PyTorch, I got a little bit confused about why they have two here, but we're going to explore that anyway. So what is a logit? So if you search what is a logit, you'll get this and you'll get statistics
10:01:26
and you'll get the log odds formula. In fact, if you want to read more, I would highly encourage it. So you could go through all of this. We're going to practice writing code for it instead. Luckily PyTorch does this for us, but logit is kind of confusing in deep learning. So if we go what is a logit in deep learning, it kind of means a different thing. It's kind of just a name of what yeah, there we go. What is the word logits in TensorFlow? As I said, TensorFlow is another
10:01:55
deep learning framework. So let's close this. What do we got? We've got a whole bunch of definitions here. Logits layer. Yeah. This is one of my favorites. In context of deep learning, the logits layer means the layer that feeds into the softmax. So softmax is a form of activation. We're going to see all of this later on because this is just words on a page right now. Softmax or other such normalization. So the output of the softmax are the probabilities for the
10:02:22
classification task and its input is the logit's layer. Whoa, there's a lot going on here. So let's just take a step back and get into writing some code. And for optimizers, I'm just going to complete this. And for optimizers, two of the most common and useful are SGD and Adam. However, PyTorch has many built in options. And as you start to learn more about the world of machine learning, you'll find that if you go to torch.optim or torch.nn. So if we have.nn, what do we have in here?
10:03:04
Loss functions. There we go. Beautiful. That's what we're after. L1 loss, which is MAE, MSC loss, cross entropy loss, CTC loss, all of these different types of loss here will depend on the problem you're working on. But I'm here to tell you that for regression and classification, two of the most common of these. See, this is that confusion again. BCE loss, BCE with logit's loss. What the hell is a logit? My goodness. Okay, that's enough. And Optim,
10:03:30
these are different optimizers. We've got probably a dozen or so here. Algorithms.
10:03:35
Add a delta, add a grad. Adam, this can be pretty full on when you first get started. But for now, just stick with SGD and the atom optimizer. They're two of the most common. Again, they may not perform the best on every single problem, but they will get you fairly far just knowing those. And then you'll pick up some of these extra ones as you go. But let's just get rid of all of, maybe we'll, so I'll put this in here, this link. So we'll create our loss function. For the loss
10:04:05
function, we're going to use torch.nn.bce with logit's loss. This is exciting. For more on what
10:04:21
binary cross entropy, which is BCE, a lot of abbreviations in machine learning and deep learning is check out this article. And then for a definition on what a logit is, we're going to see a logit in a second in deep learning. Because again, deep learning is one of those fields, a machine learning, which likes to be a bit rebellious, you know, likes to be a bit different from the pure mathematics type of fields and statistics in general. It's this beautiful
10:04:53
gestaltism and for different optimizers, see torch dot opt in. But we've covered a few of these things before. And finally, I'm going to put up here, and then for some common choices of loss functions and optimizers. Now, don't worry too much. This is why I'm linking all of these extra resources. A lot of this is covered in the book. So as we just said, set up loss function, optimizer, we just talked about these things. But I mean, you can just go to this book website
10:05:27
and reference it. Oh, we don't want that. We want this link. Come on, I knew you can't even copy and paste. How are you supposed to code? I know I've been promising code this whole time, so let's write some. So let's set up the loss function. What did we say it was? We're going to call it L O double S F N for loss function. And we're going to call B C E with logit's loss. So B C E with logit's loss. This has the sigmoid activation function built in. And we haven't covered what
10:06:01
the sigmoid activation function is, but we are going to don't you worry about that built in. In fact, if you wanted to learn what the sigmoid activation function is, how could you find out sigmoid activation function? But we're going to see it in action. Activation functions in neural networks. This is the beautiful thing about machine learning. There's so much stuff out there. People have written some great articles. You've got formulas here. PyTorch has implemented that
10:06:26
behind the scenes for us. So thank you, PyTorch. But if you recall, sigmoid activation function built in, where did we discuss the architecture of a classification network? What do we have here? Right back in the zeroth chapter of this little online book thing that we heard here. Binary classification. We have output activation. Oh, oh, look at that. So sigmoid torch dot sigmoid and pytorch. All right. And then for multi class classification, we need the softmax. Okay. Names
10:06:59
on a page again, but this is just a reference table so we can keep coming back to. So let's just keep going with this. I just want to highlight the fact that nn dot BCE loss also exists. So this requires BCE loss equals requires inputs to have gone through the sigmoid activation function
10:07:24
prior to input to BCE loss. And so let's look up the documentation. I'm going to comment that out because we're going to stick with using this one. Now, why would we stick with using this one? Let's check out the documentation, hey, torch dot nn. And I realized this video is all over the place, but we're going to step back through BCE loss with logits. Did I even say this right?
10:07:49
With logits loss. So with I got the width around the wrong way. So let's check this out. So this loss combines a sigmoid layer with the BCE loss in one single class. So if we go back to the code, BCE loss is this. So if we combined an n dot sequential, and then we passed in an n dot sigmoid, and then we went and then dot BCE loss, we'd get something similar to this. But if we keep reading in the documentation, because that's just I just literally read that it combines sigmoid with BCE
10:08:27
loss. But if we go back to the documentation, why do we want to use it? So this version is more numerically stable than using a plain sigmoid by a BCE loss, followed by a BCE loss. As by combining the operations into one layer, we take advantage of the log sum x trick for numerical stability, beautiful. So if we use this log function, loss function for our binary cross entropy, we get some numeric stability. Wonderful. So there's our loss function. We've got the sigmoid
10:08:59
activation function built in. And so we're going to see the difference between them later on, like in the flesh, optimizer, we're going to choose, hmm, let's stick with SGD, hey, old faithful stochastic gradient descent. And we have to set the parameters here, the parameters parameter params equal to our model parameters would be like, hey, stochastic gradient descent, please update. If we get another code cell behind here, please update our model parameters model
10:09:32
with respect to the loss, because we'd like our loss function to go down. So these two are going to work in tandem again, when we write our training loop, and we'll set our learning rate to 0.1. We'll see where that gets us. So that's what the optimizer is going to do. It's going to optimize all of these parameters for us, which is amazing. And the principal would be the same, even if there was 100 layers here, and 10,000, a million different parameters here. So we've got a loss function,
10:09:59
we've got an optimizer. And how about we create an evaluation metric. So let's calculate accuracy at the same time. Because that's very helpful with classification problems is accuracy. Now, what is accuracy? Well, we could look up formula for accuracy. So true positive over true positive plus true negative times 100. Okay, let's see if we can implement something similar to that just using pure pytorch. Now, why would we want accuracy? Because the accuracy is out of 100
10:10:32
examples. What percentage does our model get right? So for example, if we had a coin toss, and we did 100 coin tosses in our model predicted heads 99 out of 100 times, and it was right every single time, it might have an accuracy of 99%, because it got one wrong. So 99 out of 100, it gets it right. So dev accuracy FN accuracy function, we're going to pass it y true. So remember, any type of evaluation function or loss function is comparing the predictions to
10:11:12
the ground truth labels. So correct equals, this is going to see how many of our y true or y threads are correct. So torch equal stands for, hey, how many of these samples y true are equal to y pred? And then we're going to get the sum of that, and we need to get the item from that because we want it as a single value in Python. And then we're going to calculate the accuracy, ACC stands for accuracy, equals correct, divided by the length of samples that we have
10:11:45
as input. And then we're going to times that by 100, and then return the accuracy. Wonderful. So we now have an accuracy function, we're going to see how all the three of these come into play when we write a training loop, which we might as we get started on the next few videos, hey, I'll see you there. Welcome back. In the last video, we discussed some different loss function options for our classification models. So we learned that if we're working with binary cross
10:12:15
entropy or binary classification problems, we want to use binary cross entropy. And pie torch has two different times of binary cross entropy, except one is a bit more numerically stable. That's the BCE with logit's loss, because it has a sigmoid activation function built in. So that's straight from the pie to its documentation. And that for optimizer wise, we have a few different choices as well. So if we check out this section here on the pie torch book, we have a
10:12:42
few different loss functions and optimizers for different problems and the pie torch code that we can implement. But the premise is still the same across the board of different problems. The loss function measures how wrong our model is. And the goal of the optimizer is to optimize the model parameters in such a way that the loss function goes down. And we also implemented our own accuracy function metric, which is going to evaluate our models predictions using accuracy
10:13:13
as an evaluation metric, rather than just loss. So let's now work on training a model. So what should we do first? Well, do you remember the steps in a pie torch training loop? So to train our model, we're going to need to build a training loop. So if you watch the video on the pie torch, so if you can Google unofficial pie torch song, you should find my, there we go, the unofficial pie torch optimization loop song. We're not going to watch that. That's going to
10:13:51
be a little tidbit for the steps that we're going to code out. But that's just a fun little jingle to remember these steps here. So if we go into the book section, this is number three train model, exactly where we're up to here. But we have pie torch training loop steps. Remember, for an epoch in a range, do the forward pass, calculate the loss, optimizer zero grand, loss backward, optimizer step, step, step. We keep singing this all day. You could keep reading those steps all
10:14:22
day, but it's better to code them. But let's write this out. So forward pass to calculate the loss, three, optimizer zero grad, four. What do we do? Loss backward. So back propagation, I'll just write that up in here back propagation. We've linked to some extra resources. If you'd like to find out what's going on in back propagation, we're focused on code here, and then gradient descent. So optimizer step. So build a training loop with the following steps. However, I've kind
10:15:06
of mentioned a few things that need to be taken care of before we talk about the forward pass. So we've talked about logits. We looked up what the hell is a logit. So if we go into this stack overflow answer, we saw machine learning, what is a logit? How about we see that? We need to do a few steps. So I'm going to write this down. Let's get a bit of clarity about us, Daniel. We're kind of all over the place at the moment, but that's all right. That's the exciting part
10:15:30
of machine learning. So let's go from going from raw logits to prediction probabilities to prediction labels. That's what we want. Because to truly evaluate our model, we want to so let's write in here our model outputs going to be raw logit. So that's the definition of a logit in machine learning and deep learning. You might read some few other definitions, but for us, the raw outputs of our model, model zero are going to be referred to as logits. So then model zero,
10:16:07
so whatever comes out of here are logits. So we can convert these logits into prediction probabilities
10:16:20
by passing them to some kind of activation function, e.g. sigmoid for binary cross entropy and softmax for multi-class classification. I've got binary class e-fication. I have to sound it out every time I spell it for binary classification. So class e-fication. So we're going to see multi-class classification later on, but we want some prediction probabilities. We're going to see what they look like in a minute. So we want to go from logits to prediction
10:17:02
probabilities to prediction labels. Then we can convert our model's prediction probabilities to
10:17:15
prediction labels by either rounding them or taking the argmax. So round is for binary classification and argmax will be for the outputs of the softmax activation function, but let's see it in action first. So I've called the logits are the raw outputs of our model with no activation function. So view the first five outputs of the forward pass on the test data. So of course, our model is still instantiated with random values. So we're
10:17:56
going to set up a variable here, y logits, and model zero, we're going to pass at the test data. So x test, not text, two device, because our model is currently on our CUDA device and we need our test data on the same device or target device. Remember, that's why we're writing device agnostic codes. So this would work regardless of whether there's a GPU active or not. Let's have a look at the logits. Oh, okay. Right now, we've got some positive values here. And we can see that
10:18:27
they're on the CUDA device. And we can see that they're tracking gradients. Now, ideally, we would have run torch dot inference mode here, because we're making predictions. And the rule of thumb is whenever you make predictions with your model, you turn it into a vowel mode. We just have to remember to turn it back to train when we want to train and you run torch dot inference mode. So we get a very similar set up here. We just don't have the gradients being
10:18:53
tracked anymore. Okay. So these are called logits. The logits are the raw outputs of our model, without being passed to any activation function. So an activation function is something a little separate from a layer. So if we come up here, we've used layer. So in the neural networks that we start to build and the ones that you'll subsequently build are comprised of layers and activation functions, we're going to make the concept of an activation function a little bit more clear later
10:19:24
on. But for now, just treat it all as some form of mathematical operation. So if we were to pass data through this model, what is happening? Well, it's going through the linear layer. Now recall, we've seen this a few times now torch and then linear. If we pass data through a linear layer, it's applying the linear transformation on the incoming data. So it's performing this mathematical operation behind the scenes. So why the output equals the input x multiplied by a
10:19:53
weight tensor a this could really be w which is transposed so that this is doing a dot product plus a bias term here. And then if we jump into our model state deck, we've got weight and we've got bias. So that's the formula that's happening in these two layers. It will be different depending on the layer that we choose. But for now, we're sticking with linear. And so that the raw output of our data going through our two layers, the logits is going to be this information
10:20:21
here. However, it's not in the same format as our test data. And so if we want to make a comparison to how good our model is performing, we need apples to apples. So we need this in the same format as this, which is not of course. So we need to go to a next step. Let's use the sigmoid. So use the sigmoid activation function on our model logits. So why are we using sigmoid? Well, recall in a binary classification architecture, the output activation is the sigmoid function here. So now
10:21:02
let's jump back into here. And we're going to create some predprobs. And what this stands for on our model logits to turn them into prediction probabilities, probabilities. So why predprobs equals torch sigmoid, why logits? And now let's have a look at why predprobs. What do we get from this? Oh, when we still get numbers on a page, goodness gracious me. But the important point now is that they've gone through the sigmoid activation function, which is now we can pass these
10:21:37
to a torch dot round function. Let's have a look at this torch dot round. And what do we get? Predprobs. Oh, the same format as what we've got here. Now you might be asking like, why don't we just put torch dot round here? Well, that's a little, this step is required to, we can't just do it on the raw logits. We need to use the sigmoid activation function here to turn it into prediction probabilities. And now what is a prediction probability? Well, that's a value usually between 0 and 1
10:22:10
for how likely our model thinks it's a certain class. And in the case of binary cross entropy, these prediction probability values, let me just write this out in text. So for our prediction probability values, we need to perform a range style rounding on them. So this is a decision boundary. So this will make more sense when we go why predprobs, if it's equal to 0.5 or greater than 0.5, we set y equal to one. So y equal one. So class one, whatever that is, a red dot or a
10:22:59
blue dot, and then why predprobs, if it is less than 0.5, we set y equal to zero. So this is class zero. You can also adjust this decision boundary. So say, if you wanted to increase this value, so anything over 0.7 is one. And below that is zero. But generally, most commonly, you'll find it split at 0.5. So let's keep going. Let's actually see this in action. So how about we recode this? So find the predicted probabilities. And so we want no, sorry, we want the predicted
10:23:47
labels, that's what we want. So when we're evaluating our model, we want to convert the outputs of our model, the outputs of our model are here, the logits, the raw outputs of our model are logits. And then we can convert those logits to prediction probabilities using the sigmoid function on the logits. And then we want to find the predicted labels. So we go raw logits output of our model, prediction probabilities after passing them through an activation function, and then prediction labels.
10:24:19
This is the steps we want to take with the outputs of our model. So find the predicted labels. Let's go in here a little bit different to our regression problem previously, but nothing we can't handle. Torch round, we're going to go y-pred-probs. So I like to name it y-pred-probs for prediction probabilities and y-preds for prediction labels. Now let's go in full if we wanted to. So y-pred labels equals torch dot round torch dot sigmoid. So sigmoid activation function for binary cross
10:24:54
entropy and model zero x test dot two device. Truly this should be within inference mode code, but for now we'll just leave it like this to have a single example of what's going on here. Now I just need to count one, two, there we want. That's where we want the index. We just want it on five examples. So check for equality. And we want print torch equal. We're going to check y-pred's dot squeeze is equal to y-pred labels. So just we're doing the exact same thing. And we
10:25:34
need squeeze here to get rid of the extra dimension that comes out. You can try doing this without squeeze. So get rid of extra dimension once again. We want y-pred's dot squeeze. Fair bit of code there, but this is what's happened here. We create y-pred's. So we turn the y-pred probes into y-pred's. And then we just do the full step over again. So we make predictions with our model, we get the raw logits. So this is logits to pred probes to pred labels. So the raw
10:26:13
outputs of our model are logits. We turn the logits into prediction probabilities using torch sigmoid. And we turn the prediction probabilities into prediction labels using torch dot round. And we fulfill this criteria here. So everything above 0.5. This is what torch dot round does. Turns it into a 1. Everything below 0.5 turns it into a 0. The predictions right now are going to be quite terrible because our model is using random numbers. But y-pred's found with the steps above
10:26:44
is the same as y-pred labels doing the more than one hit. Thanks to this check for equality using torch equal y-pred's dot squeeze. And we just do the squeeze to get rid of the extra dimensions. And we have out here some labels that look like our actual y-test labels. They're in the same format, but of course they're not the same values because this model is using random weights to make predictions.
10:27:10
So we've done a fair few steps here, but I believe we are now in the right space to start building a training a test loop. So let's write that down here 3.2 building a training and testing loop. You might want to have a go at this yourself. So we've got all the steps that we need to do the forward pass. But the reason we've done this step here, the logits, then the pred probes and the pred labels, is because the inputs to our loss function up here, this requires, so BCE with
10:27:43
logits loss, requires what? Well, we're going to see that in the next video, but I'd encourage you to give it a go at implementing these steps here. Remember the jingle for an epoch in a range, do the forward pass, calculate the loss, which is BC with logits loss, optimise a zero grad, which is this one here, last backward, optimise a step, step, step. Let's do that together in the next video. Welcome back. In the last few videos, we've been working through creating a model for
10:28:17
a classification problem. And now we're up to training a model. And we've got some steps here, but we started off by discussing the concept of logits. Logits are the raw output of the model, whatever comes out of the forward functions of the layers in our model. And then we discussed how we can turn those logits into prediction probabilities using an activation function, such as sigmoid for binary classification, and softmax for multi class classification.
10:28:44
We haven't seen softmax yet, but we're going to stick with sigmoid for now because we have binary classification data. And then we can convert that from prediction probabilities to prediction labels. Because remember, when we want to evaluate our model, we want to compare apples to apples. We want our models predictions to be in the same format as our test labels. And so I took a little break after the previous video. So my collab notebook has once again
10:29:12
disconnected. So I'm just going to run all of the cells before here. It's going to reconnect up here. We should still have a GPU present. That's a good thing about Google collab is that if you change the runtime type to GPU, it'll save that wherever it saves the Google collab notebook, so that when you restart it, it should still have a GPU present. And how can we check that, of course, while we can type in device, we can run that cell. And we can also check
10:29:42
Nvidia SMI. It'll tell us if we have an Nvidia GPU with CUDA enabled ready to go. So what's our device? CUDA. Wonderful. And Nvidia SMI. Excellent. I have a Tesla P100 GPU. Ready to go. So with that being said, let's start to write a training loop. Now we've done this before, and we've got the steps up here. Do the forward pass, calculate the loss. We've spent enough on this. So we're just going to start jumping into write code. There is a little tidbit in this one,
10:30:14
though, but we'll conquer that when we get to it. So I'm going to set a manual seed,
10:30:20
torch top manual seed. And I'm going to use my favorite number 42. This is just to ensure reproducibility, if possible. Now I also want you to be aware of there is also another form of random seed manual seed, which is a CUDA random seed. Do we have the PyTorch? Yeah, reproducibility. So torch dot CUDA dot manual seed dot seed. Hmm. There is a CUDA seed somewhere. Let's try and find out. CUDA. I think PyTorch have just had an upgrade to
10:31:02
their documentation. Seed. Yeah, there we go. Okay. I knew it was there. So torch dot CUDA dot manual seed. So if we're using CUDA, we have a CUDA manual seed as well. So let's see what happens if we put that to watch that CUDA dot manual seed 42. We don't necessarily have to put these. It's just to try and get as reproducible as numbers as possible on your screen and my screen. Again, what is more important is not necessarily the numbers exactly being the same lining up
10:31:31
between our screens. It's more so the direction of which way they're going. So let's set the number of epochs. We're going to train for 100 epochs. epochs equals 100. But again, as you might have guessed, the CUDA manual seed is for if you're doing operations on a CUDA device, which in our case, we are. Well, then perhaps we'd want them to be as reproducible as possible. So speaking of CUDA devices, let's put the data to the target device because we're working with or we're writing
10:32:00
data agnostic code here. So I'm going to write x train y train equals x train two device, comma y train dot two device, that'll take care of the training data. And I'm going to do the same for the testing data equals x test two device. Because if we're going to run our model
10:32:20
on the CUDA device, we want our data to be there too. And the way we're writing our code, our code is going to be device agnostic. Have I said that enough yet? So let's also build our training and evaluation loop. Because we've covered the steps in here before, we're going to start working a little bit faster through here. And don't worry, I think you're up to it. So for an epoch in a range of epochs, what do we do? We start with training. So let me just write this.
10:32:48
Training model zero dot train. That's the model we're working with. We call the train, which is the default, but we're going to do that anyway. And as you might have guessed, the code that we're writing here is, you can functionize this. So we're going to do this later on. But just for now, the next couple of videos, the next module or two, we're going to keep writing out the training loop in full. So this is the part, the forward pass, where there's a
10:33:14
little bit of a tidbit here compared to what we've done previously. And that is because we're outputting raw logits here, if we just pass our data straight to the model. So model zero x train. And we're going to squeeze them here to get rid of an extra dimension. You can try to see what the output sizes look like without squeeze. But we're just going to call squeeze here. Remember, squeeze removes an extra one dimension from a tensor. And then to convert it
10:33:39
into prediction labels, we go torch dot round. And torch dot sigmoid, because torch dot sigmoid is an activation function, which is going to convert logits into what convert the logits into prediction probabilities. So why logits? And I'm just going to put a note here. So this is going to go turn logits into pred probes into pred labels. So we've done the forward pass. So that's a little tidbit there. We could have done all of this in one step, but I'll show you
10:34:12
why we broke this apart. So now we're going to calculate loss slash accuracy. We don't necessarily have to calculate the accuracy. But we did make an accuracy function up here. So that we can calculate the accuracy during training, we could just stick with only calculating the loss. But sometimes it's cool to visualize different metrics loss plus a few others while your model is training. So let's write some code to do that here. So we'll start off by going loss equals loss
10:34:44
f n and y logits. Ah, here's the difference of what we've done before. Previously in the notebook zero one, up to zero two now, we passed in the prediction right here. But because what's our loss function? Let's have a look at our loss function. Let's just call that see what it returns. BCE with logits loss. So the BCE with logits expects logits as input. So as you might have guessed, loss function without logits. If we had nn dot BCE loss, notice how we haven't got with logits.
10:35:25
And then we called loss f n, f n stands for function, by the way, without logits. What do we get? So BCE loss. So this loss expects prediction probabilities as input. So let's write some code to differentiate between these two. As I said, we're going to be sticking with this one. Why is that because if we look up torch BCE with logits loss, the documentation states that it's more numerically stable. So this loss combines a sigmoid layer and the BCE loss into one single
10:35:59
class, and is more numerically stable. So let's come back here, we'll keep writing some code.
10:36:06
And the accuracy is going to be accuracy f n. So our accuracy function, there's a little bit of a difference here is why true equals y train for the training data. So this will be the training accuracy. And then we have y pred equals y pred. So this is our own custom accuracy function that we wrote ourselves. This is a testament to the Pythonic nature of PyTorch as well. We've just got a pure Python function that we've slotted into our training loop,
10:36:37
which is essentially what the loss function is behind the scenes. Now, let's write here, and then dot BCE with logits loss expects raw logits. So the raw output of our model as input. Now, what if we were using a BCE loss on its own here? Well, let's just write some code for that. So let's call loss function. And then we want to pass in y pred. Or we can just go why or torch sigmoid. So why would we pass in torch sigmoid on the logits here? Because
10:37:14
remember, calling torch dot sigmoid on our logits turns our logits into prediction probabilities. And then we would pass in y train here. So if this was BCE loss expects this expects prediction probabilities as input. So does that make sense? That's the difference between with logits. So our loss function requires logits as input. Whereas if we just did straight up BCE loss, we need to call torch dot sigmoid on the logits because it expects prediction probabilities as
10:37:52
input. Now, I'm going to comment that out because our loss function is BCE with logits loss. But just keep that in mind. For some reason, you stumble across some pytorch code that's using BCE loss, not BCE with logits loss. And you find that torch dot sigmoid is calling here, or you come across some errors, because your inputs to your loss function are not what it expects. So with that being said, we can keep going with our other steps. So we're up to optimizer zero grad. So
10:38:23
optimizer dot zero grad. Oh, this is step three, by the way. And what's after this? Once we've zero grad the optimizer, we do number four, which is loss backward. We can go last backward. And then we go what's next? Optimizer step step step. So optimizer dot step. And I'm singing the unofficial pytorch optimization loop song there. This is back propagation. Calculate the gradients with respect to all of the parameters in the model. And the optimizer step is update the parameters to reduce
10:39:01
the gradients. So gradient descent, hence the descent. Now, if we want to do testing, well, we know what to do here now, we go model zero, what do we do? We call model dot of al when we're testing. And if we're making predictions, that's what we do when we test, we make predictions on the test data set, using the patterns that our model has learned on the training data set, we turn on inference mode, because we're doing inference, we want to do the
10:39:29
forward pass. And of course, we're going to compute the test logits, because logits are the raw output of our model with no modifications. X test dot squeeze, we're going to get rid of an extra one dimension there. Then we create the test pred, which is we have to do a similar calculation to what we've done here for the test pred, which is torch dot round. For our binary classification, we want prediction probabilities, which we're going to create by calling the sigmoid function
10:39:59
on the test logits, prediction probabilities that are 0.5 or above to go to one, and prediction probabilities under 0.5 to go to level zero. So two is calculate the test loss, test loss slash accuracy. How would we do this? Well, just if we've done before, and we're going to go loss FN test logits, because our loss function, we're using what we're using BCE with logits loss, expects logits as input, where do we find that out in the documentation, of course,
10:40:34
then we come back here, test logits, we're going to compare that to the Y test labels. And then for the test accuracy, what are we going to do? We're going to call accuracy FN on Y true equals Y test, and Y pred equals test pred. Now you might be thinking, why did I switch up the order here for these? Oh, and by the way, this is important to know with logits loss.
10:41:03
So with these loss functions, the order here matters of which way you put in your parameters. So predictions come first, and then true labels for our loss function. You might be wondering why I've done it the reverse for our accuracy function, Y true and Y pred. That's just because I like to be confusing. Well, not really. It's because if we go to scikit-learn, I base a lot of my structured code of how scikit-learn structures things. The scikit-learn metrics accuracy
10:41:32
score goes Y true Y pred. So I base it off that order, because the scikit-learn metrics package is very helpful. So I've based our metric evaluation metric function off this one. Whereas PyTorch's loss function does it in the reverse order, and it's important to get these in the right order. Exactly why they do it in that order. I couldn't tell you why. And we've got one final step, which is to print out what's happening. So how about we go, we're doing a lot of epochs here, 100 epochs.
10:42:10
So we'll divide the epoch by 10 to print out every epoch or every 10th epoch, sorry. And we have a couple of different metrics that we can print out this time. So we're going to print out the epoch number epoch. And then we're going to print out the loss. So loss, how many decimal points? We'll go point five here. This is going to be the training loss. We'll also do the accuracy, which will be the training accuracy. We could write trainiac here for our variable to be a little bit,
10:42:42
make them a little bit more understandable. And then we go here, but we're just going to leave it as loss and accuracy for now, because we've got test loss over here, test loss. And we're going to do the same five decimal points here. And then we're going to go test accuracy as well. Test act dot, we'll go to for the accuracy. And because it's accuracy, we want a percentage. This is the percent out of 100 guesses. What's the percentage that our model gets right on the training
10:43:13
data and the testing data, as long as we've coded all the functions correctly. Now, we've got a fair few steps here. My challenge to you is to run this. And if there are any errors, try to fix them. No doubt there's probably one or two or maybe more that we're going to have to fix in the next video. But speaking of next videos, I'll see you there. Let's train our first classification model. Well, this is very exciting. I'll see you soon.
10:43:38
Welcome back. In the last video, we wrote a mammoth amount of code, but nothing that we can't handle. We've been through a lot of these steps. We did have to talk about a few tidbits between using different loss functions, namely the BCE loss, which is binary cross entropy loss, and the BCE with logit's loss. We discussed that the BCE loss in PyTorch expects prediction probabilities as input. So we have to convert our model's logits. Logits are the raw output of the
10:44:08
model to prediction probabilities using the torch dot sigmoid activation function. And if we're using BCE with logits loss, it expects raw logits as input as sort of the name hints at. And so we just pass it straight away the raw logits. Whereas our own custom accuracy function compares labels to labels. And that's kind of what we've been stepping through over the last few videos, is going from logits to predprobs to pred labels, because that's the ideal output of our model is
10:44:39
some kind of label that we as humans can interpret. And so let's keep pushing forward. You may have already tried to run this training loop. I don't know if it works. We wrote all this code to get them in the last video. And it's probably an error somewhere. So you ready? We're going to train our first classification model together for 100 epochs. If it all goes to plan in three, two, one, let's run. Oh my gosh, it actually worked the first time. I promise you, I didn't change
10:45:07
anything in here from the last video. So let's inspect what's going on. It trains pretty fast. Why? Well, because we're using a GPU, so it's going to be accelerated as much as it can anyway. And our data set is quite small. And our network is quite small. So you won't always get networks training this fast. They did 100 epochs in like a second. So the loss. Oh,
10:45:29
0.69973. It doesn't go down very much. The accuracy even starts high and then goes down.
10:45:38
What's going on here? Our model doesn't seem to be learning anything. So what would an ideal accuracy be? An ideal accuracy is 100. And what's an ideal loss value? Well, zero, because lower is better for loss. Hmm, this is confusing. And now if we go, have a look at our blue and red dots. Where's our data? So I reckon, do we still have a data frame here? How many samples do we have of each? Let's inspect. Let's do some data analysis. Where do we create a data frame here?
10:46:13
Now, circles, do we still have this instantiated circles dot label dot? We're going to call on pandas here, value counts. Is this going to output how many of each? Okay. Wow, we've got 500 of class one and 500 of class zero. So we have 500 red dots and blue dots, which means we have a balanced data set. So if we're getting, we're basically trying to predict heads or tails here. So if we're getting an accuracy of under 50%, or about 50%, if you rounded it up.
10:46:49
Our model is basically doing as well as guessing. Well, what gives? Well, I think we should get visual with this. So let's make some predictions with our model, because these are just numbers on the page. It's hard to interpret what's going on. But our intuition now is because we have 500 samples of each, or in the case of the training data set, we have 400 of each because we have 800 samples in the training data set. And we have in the testing data set, we have 200 total
10:47:15
samples. So we have 100 of each. We're basically doing a coin flip here. Our model is as good as guessing. So turn to investigate why our model is not learning. And one of the ways we can do that is by visualizing our predictions. So let's write down here from the metrics. It looks like our model isn't learning anything. So to inspect it, let's make some predictions and make them visual. And we're right down here. In other words, visualize, visualize, visualize. All right.
10:48:00
So we've trained a model. We've at least got the structure for the training code here. But this is the right training code. We've written this code before. So you know that this set up for training code does allow a model to train. So there must be something wrong with either how we've built our model, the data set. But let's keep going and investigate together. So to do so, I've got a function that I've pre-built earlier. Did I mention that we're learning side
10:48:24
by side of a machine learning cooking show? So this is an ingredient I prepared earlier, a part of a dish. So to do so, we're going to import a function called plot decision, or maybe I'll turn this into code, plot decision boundary.
10:48:44
Welcome to the cooking show, cooking with machine learning. What model will we cook up today? So if we go to pytorch deep learning, well, it's already over here, but this is the home repo for the course, the link for this will be scattered everywhere. But there's a little function here called helper functions dot py, which I'm going to fill up with helper functions throughout the course. And this is the one I'm talking about here, plot decision boundary. Now we could just
10:49:13
copy this into our notebook, or I'm going to write some code to import this programmatically, so we can use other functions from in here. Here's our plot predictions function that we made in the last section, zero one, but this plot decision boundary is a function that I got inspired by to create from madewithml.com. Now this is another resource, a little bit of an aside, I highly recommend going through this by Goku Mohandas. It gives you the foundations of neural
10:49:41
networks and also ml ops, which is a field, which is based on getting your neural networks and machine learning models into applications that other people can use. So I can't recommend this resource enough. So please, please, please check that out if you want another resource for machine learning, but this is where this helper function came from. So thank you, Goku Mohandas. I've made a little bit of modifications for this course, but not too many. So we could either copy that, paste it in
10:50:12
here, or we could write some code to import it for us magically, or using the power of the internet, right, because that's what we are. We're programmers, we're machine learning engineers, we're data scientists. So from pathlib, so the request module in Python is a module that allows you to make requests, a request is like going to a website, hey, I'd like to get this code from you, or this information from you, can you please send it to me? So that's what that allows us to do,
10:50:40
and pathlib, we've seen pathlib before, but it allows us to create file parts. Because why? Well, we want to save this helper function dot pi script to our Google collab files. And so we can do this with a little bit of code. So download helper functions from learn pytorch repo. If it's not already downloaded. So let's see how we can do that. So we're going to write some if else code to check to see if the path of helper functions dot pi already exist, we don't want to download it again.
10:51:23
So at the moment, it doesn't exist. So this if statement is going to return false. So let's just print out what it does if it returns true helper functions dot pi already exists. We might we could even probably do a try and accept looping about if else will help us out for now. So if it exists else, print downloading helper functions dot pi. So ours doesn't exist. So it's going to make a request or let's set up our request request dot get. And here's where we can put in a URL. But we
10:52:02
need the raw version of it. So this is the raw version. If we go back, this is just pytorch deep learning the repo for this course slash helper functions. If I click raw, I'm going to copy that. Oh, don't want to go in there want to go into request get type that in this has to be in a string format. So we get the raw URL. And then we're going to go with open, we're going to open a file called helper functions dot pi. And we're going to set the context to be right binary,
10:52:39
which is wb as file F is a common short version of writing file. Because we're going to call file dot write, and then request dot content. So this code is basically saying hey requests, get the information that's at this link here, which is of course, all of this code here, which is a Python script. And then we're going to create a file called helper functions dot pi, which gives us write permissions. We're going to name it F, which is short for file. And then
10:53:09
we're going to call on it file dot write the content of the request. So instead of talking through it, how about we see it in action? We'll know if it works if we can from helper functions import plot predictions, we're going to use plot predictions later on, as well as plot decision boundary. So plot predictions we wrote in the last section. Wonderful. I'm going to write here, downloading helper functions dot pi did at work. We have helper functions dot pi. Look at that,
10:53:42
we've done it programmatically. Can we view this in Google column? Oh my goodness, yes we can. And look at that. So this may evolve by the time you do the course, but these are just some general helper functions rather than writing all of this out. If you would like to know what's going on in plot decision boundary, I encourage you to read through here. And what's going on, you can step by step at yourself. There is nothing here that you can't tackle yourself. It's all
10:54:07
just Python code, no secrets just Python code. We've got we're making predictions with a PyTorch model. And then we're testing for multi class or binary. So we're going to get out of that. But now let's see the ultimate test is if the plot decision boundary function works. So again, we could discuss plot decision boundary of the model. We could discuss what it does behind the scenes
10:54:33
to the cows come home. But we're going to see it in real life here. I like to get visual. So fig size 12, six, we're going to create a plot here, because we are adhering to the data explorer's motto of visualize visualize visualize. And we want to subplot because we're going to compare our training and test sets here, train. And then we're going to go PLT, or actually we'll plot the first one, plot decision boundary. Now, because we're doing a training plot here,
10:55:05
we're going to pass in model zero and X train and Y train. Now, this is the order that the parameters go in. If we press command shift space, I believe Google collab, if it's working with me, we'll put up a doc string. There we go, plot decision boundary. Look at the inputs that it takes model, which is torch and end up module. And we've got X, which is our X value, which is a torch tensor, and Y, which is our torch tensor value here. So that's for the training data.
10:55:36
Now, let's do the same for the testing data, plot dot subplot. This is going to be one, two, two for the index. This is just number of rows of the plot, number of columns. And this is the index. So this plot will appear on the first slot. We're going to see this anyway. Anything below this code will appear on the second slot, PLT dot title. And we're going to call this one test. Then we're going to call plot decision boundary. If this works, this is going to be some
10:56:04
serious magic. I love visualization functions in machine learning. Okay, you ready? Three, two, one, let's check it out. How's our model doing? Oh, look at that. Oh, now it's clear. So behind the scenes, this is the plots that plot decision boundary is making. Of course, this is the training data. This is the testing data, not as many dot points here, but the same sort of line of what's going on. So this is the line that our model is trying to draw through the
10:56:33
data. No wonder it's getting about 50% accuracy and the loss isn't going down. It's just trying to split the data straight through the middle. It's drawing a straight line. But our data is circular. Why do you think it's drawing a straight line? Well, do you think it has anything to do with the fact that our model is just made with using pure linear layers? Let's go back to our model. What's it comprised on? Just a couple of linear layers. What's a linear line? If we look up linear
10:57:01
line, is this going to work with me? I don't actually think it might. There we go. Linear line, all straight lines. So I want you to have a think about this, even if you're completely new to deep learning, can we? You can answer this question. Can we ever separate this circular data with straight lines? I mean, maybe we could if we drew straight lines here, but then trying to curve them around. But there's an easier way. We're going to see that later on. For now,
10:57:29
how about we try to improve our model? So the model that we built, we've got 100 epochs. I wonder if our model will improve if we trained it for longer. So that's a little bit of a challenge before the next video. See if you can train the model for 1000 epochs. Does that improve the results here? And if it doesn't improve the results here, have a think about why that might be.
10:57:52
I'll see you in the next video. Welcome back. In the last video, we wrote some code to download a series of helper functions from our helper functions dot pi. And later on, you'll see why this is quite standard practice as you write more and more code is to write some code, store them somewhere such as a Python script like this. And then instead of us rewriting everything that we have and helper functions, we just import them and then use them later on. This is similar to
10:58:20
what we've been doing with PyTorch. PyTorch is essentially just a collection of Python scripts that we're using to build neural networks. Well, there's a lot more than what we've just done. I mean, we've got one here, but PyTorch is a collection of probably hundreds of different Python scripts. But that's beside the point. We're trying to train a model here to separate blue and red dots. But our current model is only drawing straight lines. And I got you to
10:58:45
have a think about whether our straight line model, our linear model could ever separate this data. Maybe it could. And I issued the challenge to see if it could if you trained for 1000 epochs. So did it improve at anything? Is the accuracy any higher? Well, speaking of training for more epochs, we're up to section number five, improving a model. This is from a model perspective. So now let's discuss some ways. If you were getting results after you train a machine learning model or a
10:59:20
deep learning model, whatever kind of model you're working with, and you weren't happy with those results. So how could you go about improving them? So this is going to be a little bit of an overview of what we're going to get into. So one way is to add more layers. So give the model more chances to learn about patterns in the data. Why would that help? Because if our model currently has two layers, model zero dot state dinked. Well, we've got however many numbers here, 20 or so. So this
10:59:56
is zero flayer. This is the first layer. If we had 10 of these, well, we'd have 10 times the amount of parameters to try and learn the patterns in this data, a representation of this data. Another way is to add more hidden units. So what I mean by that is we created this model here, and each of these layers has five hidden units. The first one outputs, out features equals five, and this one takes in features equals five. So we could go from, go from five hidden units to
11:00:31
10 hidden units. The same principle as above applies here is that the more parameters our model has to represent our data, the potentially now I say potentially here because some of these things might not necessarily work. So our data sets quite simple. So maybe if we added too many layers, our models trying to learn things that are too complex, it's trying to adjust too many numbers for the data set that we have the same thing for more hidden units. What other options do we
11:01:01
have? Well, we could fit for longer, give the model more of a chance to learn because every epoch is one pass through the data. So maybe 100 times looking at this data set wasn't enough. So maybe you could fit for 1000 times, which was the challenge. Then there's change in the activation functions, which we're using sigmoid at the moment, which is generally the activation function you use for a binary classification problem. But there are also activation functions
11:01:28
you can put within your model. Hmm, there's a little hint that we'll get to that later. Then there's change the learning rate. So the learning rate is the amount the optimizer will adjust these every epoch. And if it's too small, our model might not learn anything because it's taking forever to change these numbers. But if also on the other side of things, if the learning rate is too high, these updates might be too large. And our model might just explode. There's an
11:01:58
actual problem in machine learning called exploding gradient problem, where the numbers just get too large. On the other side, there's also a vanishing gradients problem, where the gradients just go basically to zero too quickly. And then there's also change the loss function. But I feel like for now, sigmoid and binary cross entropy, pretty good, pretty standard. So we're going to have a look at some options here, add more layers and fit for longer, maybe changing the learning
11:02:28
rate. But let's just add a little bit of color to what we've been talking about. Right now, we've fit the model to the data and made a prediction. I'm just going to step through this. Where are we up to? We've done this, we've done this, we've done these two, we've built a training loop, we've fit the model to the data, made a prediction, we've evaluated our model visually, and we're not happy with that. So we're up to number five, we're going to improve through
11:02:50
experimentation. We don't need to use TensorBoard just yet, we're going to talk about this as our high level. TensorBoard is a tool or a utility from PyTorch, which helps you to monitor experiments. We'll see that later on. And then we'll get to this, we won't save our model until we've got one that we're happy with. And so if we look at what we've just talked about improving a model from a model's perspective, let's talk about the things we've talked about with some color this time. So
11:03:16
say we've got a model here, this isn't the exact model that we're working with, but it's similar structure. We've got one, two, three, four layers, we've got a loss function BC with Logit's loss, we've got an optimizer, optimizer stochastic gradient descent, and if we did write some training code, this is 10 epochs. And then the testing code here, I've just cut it out because it wouldn't fit on the slide. Then if we wanted to go to a larger model, let's add some color here so we can highlight
11:03:43
what's happening, adding layers. Okay, so this one's got one, two, three, four, five, six layers. And we've got another color here, which is I'd say this is like a little bit of a greeny blue increase the number of hidden units. Okay, so the hidden units are these features here. We've gone from 100 to 128 to 128. Remember, the out features of a previous layer have to line up with the in features of a next layer. Then we've gone to 256. Wow. So remember how I said multiples
11:04:14
of eight are pretty good generally in deep learning? Well, this is where these numbers come from. And then what else do we have change slash add activation functions? We haven't seen this before and end up relu. If you want to jump ahead and have a look at what and end up relu is, how would you find out about it? Well, I just Google and end up relu. But we're going to have a look at what this is later on. We can see here that this one's got one, but this larger model has
11:04:41
some relu's scattered between the linear layers. Hmm, maybe that's a hint. If we combine a linear layer with a relu, what's a relu layer? I'm not going to spoil this. We're going to find out later on change the optimization function. Okay. So we've got SGD. Do you recall how I said Adam is another popular one that works fairly well across a lot of problems as well. So Adam might be a better option for us here. The learning rate as well. So maybe this learning rate was a
11:05:11
little too high. And so we've divided it by 10. And then finally, fitting for longer. So instead of 10 epochs, we've gone to 100. So how about we try to implement some of these with our own model to see if it improves what we've got going on here? Because frankly, like, this isn't satisfactory. We're trying to build a neural network here. Neural networks are supposed to be these models that can learn almost anything. And we can't even separate some blue dots from
11:05:35
some red dots. So in the next video, how about we run through writing some code to do some of these steps here? In fact, if you want to try yourself, I'd highly encourage that. So I'd start with trying to add some more layers and add some more hitting units and fitting for longer. You can keep all of the other settings the same for now. But I'll see you in the next video. Welcome back. In the last video, we discussed some options to improve our model from a model perspective. And
11:06:05
namely, we're trying to improve it so that the predictions are better, so that the patterns it learns better represent the data. So we can separate blue dots from red dots. And you might be wondering why we said from a model perspective here. So let me just write these down. These options are all from a models perspective, because they deal directly with the model, rather than the data. So there's another way to improve a models results is if the model was sound already,
11:06:41
in machine learning and deep learning, you may be aware that generally if you have more data samples, the model learns or gets better results because it has more opportunity to learn. There's a few other ways to improve a model from a data perspective, but we're going to focus on improving a model from a models perspective. So, and because these options are all values we as machine learning
11:07:13
engineers and data scientists can change, they are referred to as hyper parameters. So a little bit of an important distinction here. Parameters are the numbers within a model. The parameters here, like these values, the weights and biases are parameters, are the values a model updates by itself. Hyper parameters are what we as machine learning engineers and data scientists, such as adding more layers, more hidden units, fitting for longer
11:07:48
number of epochs, activation functions, learning rate, loss functions are hyper parameters because they're values that we can change. So let's change some of the hyper parameters of our model. So we'll create circle model v1. We're going to import from nn.module as well. We could write this model using nn.sequential, but we're going to subclass nn.module for practice.
11:08:15
Why would we use nn.sequential? Well, because as you'll see, our model is not too complicated, but we subclass nn.module. In fact, nn.sequential. So if we write here, nn.sequential is also a version of nn.module. But we subclass nn.module here for one for practice and for later on, if we wanted to, or if you wanted to make more complex models, you're going to see a subclass of nn.module a lot in the wild. So the first change we're going to update is the number
11:08:52
of hidden units. So out features, I might write this down before we do it. Let's try and improve our model by adding more hidden units. So this will go from five and we'll increase it to 10. And we want to increase the number of layers. So we want to go from two to three. We'll add an extra layer and then increase the number of epochs. So we're going to go from 100 to 1,000. Now, what can you, we're going to put on our scientist hats for a second. What would be the problem with
11:09:39
the way we're running this experiment? If we're doing all three things in one hit, why might that be problematic? Well, because we might not know which one offered the improvement if there is any improvement or degradation. So just to keep in mind going forward, I'm just doing this as an example of how we can change all of these. But generally, when you're doing machine learning experiments, you'd only like to change one value at a time and track the results. So that's called
11:10:10
experiment tracking and machine learning. We're going to have a look at experiment tracking a little later on in the course, but just keep that in mind. A scientist likes to change one variable of what's going on so that they can control what's happening. But we're going to create this next layer here layer two. And of course, it takes the same number of out features as in features as the previous layer. This is two because why our X train has. Let's look at just
11:10:40
the first five samples has two features. So now we're going to create self layer three, which equals an n dot linear. The in features here is going to be 10. Why? Because the layer above has out features equals 10. So what we've changed here so far is we've got hidden units previously in the zero of this model was five. And now we've got a third layout, which previously before was two. So these are two of our main changes here. And out features equals one, because why? Let's
11:11:12
have a look at speaking of why. Our why is just one number. So remember the shapes, the input and output shapes of a model is one of the most important things in deep learning. We're going to see different values for the shapes later on. But because we're working with this data set, we're focused on two in features and one out feature. So now that we've got our layers prepared, what's next? Well, we have to override the forward method, because every subclass of
11:11:41
an n dot module has to implement a forward method. So what are we going to do here? Well, we could, let me just show you one option. We could go z, which would be z for logits. Logits is actually represented by z, fun fact. But you could actually put any variable here. So this could be x one, or you could reset x if you wanted to. I just look putting a different one because it's a little less confusing for me. And then we could go update z by going self layer two. And then the,
11:12:14
because z above is the output of layer one, it now goes into here. And then if we go z,
11:12:22
again, equals self layer three, what's this going to take? It's going to take z from above. So this is saying, hey, give me x, put it through layer one, assign it to z. And then create a new variable z or override z with self layer two with z from before as the input. And then we've got z again, the output of layer two has the input for layer three. And then we could return z. So that's just passing our data through each one of these layers here. But a way that
11:12:51
you can leverage speedups in PyTorch is to call them all at once. So layer three, and we're going to put self dot layer two. And this is generally how I'm going to write them. But it also behind the scenes, because it's performing all the operations at once, you leverage whatever speed ups you can get. Oh, this should be layer one. So it goes in order here. So what's happening? Well, it's computing the inside of the brackets first. So layer one, x is going through layer one.
11:13:26
And then the output of x into layer one is going into layer two. And then the same again, for layer three. So this way, this way of writing operations, leverages, speed ups, where possible
11:13:47
behind the scenes. And so we've done our Ford method there. We're just passing our data through layers with an extra hidden units, and an extra layer overall. So now let's create an instance of circle model v one, which we're going to set to model one. And we're going to write circle model v one. And we're going to send it to the target device, because we like writing device agnostic code.
11:14:14
And then we're going to check out model one. So let's have a look at what's going on there. Beautiful. So now we have a three layered model with more hidden units. So I wonder if we trained this model for longer, are we going to get improvements here? So my challenge to you is we've already done these steps before. We're going to do them over the next couple of videos for completeness.
11:14:38
But we need to what create a loss function. So I'll give you a hint. It's very similar to the one we've already used. And we need to create an optimizer. And then once we've done that, we need to write a training and evaluation loop for model one. So give that a shot. Otherwise, I'll see you in the next video. We'll do this all together. Welcome back. In the last video, we subclassed nn.module to create circle model V one, which is an upgrade on circle model V zero. In the
11:15:15
fact that we added more hidden units. So from five to 10. And we added a whole extra layer. And we've got an instance of it ready to go. So we're up to in the workflow. We've got our data. Well, we haven't changed the data. So we've built our new model. We now need to pick a loss function. And I hinted at before that we're going to use the same loss function as before. The same optimizer. You might have already done all of these steps. So you may know whether this
11:15:41
model works on our data set or not. But that's what we're going to work towards finding out in this video. So we've built our new model. Now let's pick a loss function and optimizer. We could almost do all of this with our eyes closed now, build a training loop, fit the model to the data, make a prediction and evaluate the model. We'll come back here. And let's set up a loss function. And by the way, if you're wondering, like, why would adding more features here, we've kind of
11:16:07
hinted at this before. And why would an extra layer improve our model? Well, again, it's back to the fact that if we add more neurons, if we add more hidden units, and if we add more layers, it just gives our model more numbers to adjust. So look at what's going on here, layer one, layer two. Look how many more we have compared to model zero dot state date.
11:16:36
We have all of these. This is model zero. And we just upgraded it. Look how many more we have from just adding an extra layer and more hidden units. So now we have our optimizer can change these values to hopefully create a better representation of the data we're trying to fit. So we just have more opportunity to learn patterns in our target data set. So that's the theory behind it. So let's get rid of ease. Let's create a loss function. What are we going to use? Well,
11:17:06
we're going to use nn dot BCE with logit's loss. And our optimizer is going to be what? We're going to keep that as the same as before, torch dot opt in dot SGD. But we have to be aware that because we're using a new model, we have to pass in params of model one. These are the parameters we want to optimize. And the LR is going to be 0.1. Is that the same LR we use before learning rate? 0.1. Oh, potentially that our learning rate may be too big. 0.1. Where do we create our
11:17:42
optimizer? So we've written a lot of code here. Optimizer. There we go. 0.1. That's all right. So we'll keep it at 0.1 just to keep as many things the same as possible. So we're going to set up torch dot manual seed 42 to make training as reproducible as possible torch dot CUDA dot manual seed 42. Now, as I said before, don't worry too much if your numbers aren't exactly the same as mine. The direction is more important, whether it's good or bad direction. So now let's set up epochs.
11:18:17
We want to train for longer this time as well. So 1000 epochs. This is one of our three improvements that we're trying to do. Adding more hidden units, increase the number of layers and increase the number of epochs. So we're going to give our model 1000 looks at the data to try and improve its patterns. So put data on the target device. We want to write device agnostic code. And yes, we've already done this, but we're going to write it out again for practice because even though we
11:18:49
could functionize a lot of this, it's good while we're in still the foundation stages to practice what's going on here, because I want you to be able to do this with your eyes closed before we start to functionize it. So put the training data and the testing data to the target device, whatever it is, CPU or GPU. And then we're going to, well, what's our song? For an epoch in range.
11:19:16
Let's loop through the epochs. We're going to start off with training. What do we do for training? Well, we set model one to train. And then what's our first step? Well, we have to forward pass. What's our outputs of the model? Well, the raw outputs of a model are logits. So model one, we're going to pass it the training data. We're going to squeeze it so that we get rid of an extra one dimension. If you don't believe me that we would like to get rid of that one dimension,
11:19:41
try running the code without that dot squeeze. And why pred equals torch dot round.
11:19:48
And torch dot sigmoid, why we're calling sigmoid on our logits to go from logits to prediction probabilities to prediction labels. And then what do we do next? Well, we calculate the loss
11:20:09
slash accuracy to here. And remember, accuracy is optional, but loss is not optional. So we're going to pass in here, our loss function is going to take in. I wonder if it'll work with just straight up why pred? I don't think it will because we're using we need logits in here. Why logits and why train? Because why? Oh, Google collab correcting the wrong thing. We have why logits because we're using BCE with logits loss here. So let's keep pushing forward. We want our accuracy now,
11:20:44
which is our accuracy function. And we're going to pass in the order here, which is the reverse of above, a little confusing, but I've kept the evaluation function in the same order as scikit loan. Why pred equals y pred? Three, we're going to zero the gradients of the optimizer, optimizer zero grad. And you might notice that we've started to pick up the pace a little. That is perfectly fine. If I'm typing too fast, you can always slow down the video,
11:21:13
or you could just watch what we're doing and then code it out yourself afterwards, the code resources will always be available. We're going to take the last backward
11:21:22
and perform back propagation. The only reason we're going faster is because we've covered these steps. So anything that we sort of spend time here, we've covered in a previous video, optimizer step. And this is where the adjustments to all of our models parameters are going to take place to hopefully create a better representation of the data. And then we've got testing. What's the first step that we do in testing? Well, we call model one dot a vowel to put it in evaluation
11:21:51
mode. And because we're making predictions, we're going to turn on torch inference mode predictions. I call them predictions. Some other places call it inference.
11:22:01
Remember machine learning has a lot of different names for the same thing. Forward pass. So we're going to create the test logits here. Equals model one X test. And we're going to squeeze them because we won't don't want the extra one dimension. Just going to add some code cells here so that we have more space and I'm typing in the middle of the screen.
11:22:22
Then I'm going to put in test pred here. How do we get from logits to predictions? Well, we go torch dot round. And then we go torch dot sigmoid y sigmoid because we're working with a binary classification problem. And to convert logits from a binary classification problem to prediction probabilities, we use the sigmoid activation function. And then we're going to calculate the loss. So how wrong is our model on the test data? So test last equals loss function.
11:22:53
We're going to pass it in the test logits. And then we're going to pass it in Y test for the ideal labels. And then we're going to also calculate test accuracy. And test accuracy is going to take in Y true equals Y test. So the test labels and Y pred equals test pred. So the test predictions
11:23:17
test predictions here. And our final step is to print out what's happening. So print out what's happening. Oh, every tutorial needs a song. If I could, I'd teach everything with song. Song and dance. So because we're training for 1000 epochs, how about every 100 epochs we print out something. So print f string, and we're going to write epoch in here. So we know what epoch our models on. And then we're going to print out the loss. Of course, this is going to be the training
11:23:50
loss. Because the test loss has test at the front of it. And then accuracy here. Now, of course, this is going to be the training accuracy. We go here. And then we're going to pipe. And we're going to print out the test loss. And we want the test loss here. We're going to take this to five decimal places. Again, when we see the printouts of the different values, do not worry too much about the exact numbers on my screen appearing on your screen, because that is inherent to the
11:24:24
randomness of machine learning. So have we got the direction is more important? Have we got, we need a percentage sign here, because that's going to be a bit more complete for accuracy. Have we got any errors here? I don't know. I'm just, we've just all coded this free hand, right? There's a lot of code going on here. So we're about to train our next model, which is the biggest model we've built so far in this course, three layers, 10 hidden units on
11:24:49
each layer. Let's see what we've got. Three, two, one, run. Oh, what? What? A thousand epochs, an extra hidden layer, more hidden units. And we still, our model is still basically a coin toss. 50%. Now, this can't be for real. Let's plot the decision boundary. Plot the decision boundary. To find out, let's get a bit visual. Plot figure, actually, to prevent us from writing out all of the plot code, let's just go up here, and we'll copy this. Now, you know,
11:25:28
I'm not the biggest fan of copying code. But for this case, we've already written it. So there's nothing really new here to cover. And we're going to just change this from model zero to model one, because why it's our new model that we just trained. And so behind the scenes, plot decision boundary is going to make predictions with the target model on the target data set and put it into a nice visual representation for us. Oh, I said nice visual representation. What does this
11:25:56
look like? We've just got a coin toss on our data set. Our model is just again, it's trying to draw a straight line to separate circular data. Now, why is this? Our model is based on linear, is our data nonlinear? Hmm, maybe I've revealed a few of my tricks. I've done a couple of reveals over the past few videos. But this is still quite annoying. And it can be fairly annoying when you're training models and they're not working. So how about we verify that this model
11:26:30
can learn anything? Because right now it's just basically guessing for our data set. So this model looks a lot like the model we built in section 01. Let's go back to this. This is the learn pytorch.io book pytorch workflow fundamentals. Where did we create a model model building essentials? Where did we build a model? Linear regression model? Yeah, here. And then dot linear. But we built this model down here. So all we've changed from 01 to here is we've added
11:27:05
a couple of layers. The forward computation is quite similar. If this model can learn something on a straight line, can this model learn something on a straight line? So that's my challenge to you is grab the data set that we created in this previous notebook. So data, you could just reproduce this in exact data set. And see if you can write some code to fit the model that we built here. This one here on the data set that we created in here. Because I want to verify that
11:27:40
this model can learn anything. Because right now it seems like it's not learning anything at all. And that's quite frustrating. So give that a shot. And I'll see you in the next video. Welcome back. In the past few videos, we've tried to build a model to separate the blue from red dots yet. Our previous efforts have proven futile, but don't worry. We're going to get there. I promise you we're going to get there. And I may have a little bit of inside information here. But we're
11:28:08
going to build a model to separate these blue dots from red dots, a fundamental classification model. And we tried a few things in the last couple of videos, such as training for longer, so more epochs. We added another layer. We increased the hidden units because we learned of a few methods to improve a model from a model perspective, such as upgrading the hyperparameters, such as number of layers, more hidden units, fitting for longer, changing the activation functions,
11:28:36
changing the learning rate, we haven't quite done that one yet, and changing the loss function. One way that I like to troubleshoot problems is I'm going to put a subheading here, 5.1. We're going to prepare or preparing data to see if our model can fit a straight line.
11:28:56
So one way to troubleshoot, this is my trick for troubleshooting problems, especially neural networks, but just machine learning in general, to troubleshoot a larger problem is to test out a smaller problem. And so why is this? Well, because we know that we had something working in a previous section, so 01, PyTorch, workflow fundamentals, we built a model here that worked. And if we go right down, we know that this linear model can fit a straight line. So we're going
11:29:36
to replicate a data set to fit a straight line to see if the model that we're building here can learn anything at all, because right now it seems like it can't. It's just tossing a coin displayed between our data here, which is not ideal. So let's make some data. But yeah, this is the, let's create a smaller problem, one that we know that works, and then add more complexity to try and solve our larger problem. So create some data. This is going to be the same as notebook 01.
11:30:05
And I'm going to set up weight equals 0.7 bias equals 0.3. We're going to move quite quickly through this because we've seen this in module one, but the overall takeaway from this is we're going to see if our model works on any kind of problem at all, or do we have something fundamentally wrong, create data. We're going to call it x regression, because it's a straight line, and we want it to predict a number rather than a class. So you might be thinking, oh, we might have to change
11:30:38
a few things of our model architecture. Well, we'll see that in a second dot unsqueeze. And we're going to go on the first dimension here or dim equals one. And why regression, we're going to use the linear regression formula as well, wait times x, x regression, that is, because we're working
11:31:00
with a new data set here, plus the bias. So this is linear regression formula. Without epsilon. So it's a simplified version of linear regression, but the same formula that we've seen in a previous section. So now let's check the data. Nothing we really haven't covered here, but we're going to do a sanity check on it to make sure that we're dealing with what we're dealing with.
11:31:29
What we're dealing with is not just a load of garbage. Because it's all about the data and machine learning. I can't stress to you enough. That's the data explorer's motto is to visualize, visualize, visualize. Oh, what did we get wrong here? Unsqueeze. Did you notice that typo? Why didn't you say something? I'm kidding. There we go. Okay, so we've got 100 samples of x. We've got a different step size here, but that's all right. Let's have a little bit of fun with this. And we've got one x-value, which is, you know, a little bit more.
11:31:58
One x value per y value is a very similar data set to what we use before. Now, what do we do once we have a data set? Well, if we haven't already got training and test splits, we better make them. So create train and test splits. And then we're going to go train split. We're going to use 80% equals int 0.8 times the length of, or we could just put 100 in there.
11:32:27
But we're going to be specific here. And then we're going to go x train regression, y train regression equals. What are these equal? Well, we're going to go on x regression. And we're going to index up to the train split on the x. And then for the y, y regression, we're going to index up to the train split.
11:32:55
Wonderful. And then we can do the same on the test or creating the test data. Nothing really new here that we need to discuss. We're creating training and test sets. What do they do for each of them? Well, the model is going to hopefully learn patterns in the training data set that is able to model the testing data set. And we're going to see that in a second.
11:33:21
So if we check the length of each, what do we have? Length x train regression. We might just check x train x test regression. What do we have here? And then we're going to go length y train regression. Long variable names here. Excuse me for that. But we want to keep it separate from our already existing x and y data. What values do we have here?
11:33:52
80, 20, 80, 20, beautiful. So 80 training samples to 100 testing samples. That should be enough. Now, because we've got our helper functions file here. And if you don't have this, remember, we wrote some code up here before to where is it? To download it from the course GitHub, and we imported plot predictions from it. Now, if we have a look at helper functions.py, it contains the plot predictions function that we created in the last section, section 0.1. There we go. Plot predictions.
11:34:30
So we're just running this exact same function here, or we're about to run it. It's going to save us from re typing out all of this. That's the beauty of having a helper functions.py file. So if we come down here, let's plot our data to visually inspected. Right now, it's just numbers on a page. And we're not going to plot really any predictions because we don't have any predictions yet.
11:34:52
But we'll pass in the train data is equal to X train regression. And then the next one is the train labels, which is equal to Y train regression. And then we have the test data, which is equal to X test regression. And then we have the test labels. Now, I think this should be labels too. Yeah, there we go. Y test progression might be proven wrong as we try to run this function.
11:35:27
Okay, there we go. So we have some training data and we have some testing data. Now, do you think that our model model one, we have a look what's model one could fit this data. Does it have the right amount of in and out features? We may have to adjust these slightly. So I'd like you to think about that. Do we have to change the input features to our model for this data set? And do we have to change the out features of our model for this data set? We'll find out in the next video.
11:36:00
Welcome back. We're currently working through a little side project here, but really the philosophy of what we're doing. We just created a straight line data set because we know that we've built a model in the past back in section 01 to fit a straight line data set. And why are we doing this? Well, because the model that we've built so far is not fitting or not working on our circular data set here on our classification data set.
11:36:26
And so one way to troubleshoot a larger problem is to test out a smaller problem first. So later on, if you're working with a big machine learning data set, you'd probably start with a smaller portion of that data set first. Likewise, with a larger machine learning model, instead of starting with a huge model, you'll start with a small model.
11:36:46
So we're taking a step back here to see if our model is going to learn anything at all on a straight line data set so that we can improve it for a non-straight line data set. And there's another hint. Oh, we're going to cover it in a second. I promise you. But let's see how now we can adjust model one to fit a straight line. And I should do the question at the end of last video. Do we have to adjust the parameters of model one in any way shape or form to fit this straight line data?
11:37:16
And you may have realized or you may not have that our model one is set up for our classification data, which has two X input features. Whereas this data, if we go X train regression, how many input features do we have? We just get the first sample. There's only one value. Or maybe we get the first 10. There's only one value per, let's remind ourselves, this is input and output shapes, one of the most fundamental things in machine learning and deep learning.
11:37:52
And trust me, I still get this wrong all the time. So that's why I'm harping on about it. We have one feature per one label. So we have to adjust our model slightly. We have to change the end features to be one instead of two. The out features can stay the same because we want one number to come out.
11:38:08
So what we're going to do is code up a little bit different version of model one. So same architecture as model one. But using NN dot sequential, we're going to do the faster way of coding a model here. Let's create model two and NN dot sequential. The only thing that's going to change is the number of input features. So this will be the exact same code as model one. And the only difference, as I said, will be features or in features is one. And then we'll go out features equals 10.
11:38:42
So 10 hidden units in the first layer. And of course, the second layer, the number of features here has to line up with the out features of the previous layer. This one's going to output 10 features as well. So we're scaling things up from one feature to 10 to try and give our model as much of a chance or as many parameters as possible. Of course, we could make this number quite large. We could make it a thousand features if we want. But there is an upper bound on these things.
11:39:08
And I'm going to let you find those in your experience as a machine learning engineer and a data scientist. But for now, we're keeping it nice and small. So we can run as many experiments as possible. Beautiful. Look at that. We've created a sequential model. What happens with NN dot sequential? Data goes in here, passes through this layer. Then it passes through this layer. Then it passes through this layer. And what happens when it goes through the layer?
11:39:31
It triggers the layers forward method, the internal forward method. In the case of NN dot linear, we've seen it. It's got the linear regression formula. So if we go NN dot linear, it performs this mathematical operation, the linear transformation. But we've seen that before. Let's keep pushing forward. Let's create a loss and an optimizer loss and optimize. We're going to work through our workflow. So loss function, we have to adjust this slightly.
11:40:00
We're going to use the L1 loss because why we're dealing with a regression problem here rather than a classification problem. And our optimizer, what can we use for our optimizer? How about we bring in just the exact same optimizer SGD that we've been using for our classification data. So model two dot params or parameters. Always get a little bit confused. And we'll give it an LR of 0.1 because that's what we've been using so far. This is the params here.
11:40:30
So we want our optimizer to optimize our model two parameters here with a learning rate of 0.1. The learning rate is what? The amount each parameter will be or the multiplier that will be applied to each parameter each epoch. So now let's train the model. Do you think we could do that in this video? I think we can. So we might just train it on the training data set and then we can evaluate it on the test data set separately.
11:41:00
So we'll set up both manual seeds, CUDA and because we've set our model to the device up here. So it should be on the GPU or whatever device you have active. So set the number of epochs. How many epochs should we set? Well, we set a thousand before, so we'll keep it at that. epochs equals a thousand. And now we're getting really good at this sort of stuff here. Let's put our data. Put the data on the target device.
11:41:30
And I know we've done a lot of the similar steps before, but there's a reason for that. I've kept all these in here because I'd like you to buy the end of this course is to sort of know all of this stuff off by heart. And even if you don't know it all off my heart, because trust me, I don't, you know where to look.
11:41:47
So X train regression, we're going to send this to device. And then we're going to go Y train regression, just a reminder or something to get you to think while we're writing this code. What would happen if we didn't put our data on the same device as a model? We've seen that error come up before, but what happens? Well, I've just kind of given away, haven't you Daniel? Well, that was a great question. Our code will air off.
11:42:16
Oh, well, don't worry. There's plenty of questions I've been giving you that I haven't given the answer to yet. Device a beautiful. We've got a device agnostic code for the model and for the data. And now let's loop through epochs. So train. We're going to for epoch in range epochs for an epoch in a range. Do the forward pass. Calculate the loss. So Y pred equals model two. This is the forward pass. X train regression.
11:42:49
It's all going to work out hunky Dory because our model and our data are on the same device loss equals what we're going to bring in our loss function. Then we're going to compare the predictions to Y train regression to the Y labels. What do we do next? Optimize a zero grad. Optimize a dot zero grad. We're doing all of this with our comments. Look at us go. Loss backward and what's next? Optimize a step, step, step. And of course, we could do some testing here.
11:43:24
Testing. We'll go model two dot a vowel. And then we'll go with torch dot inference mode. We'll do the forward pass. We'll create the test predictions equals model two dot X test regression. And then we'll go the test loss equals loss FN on the test predictions and versus the Y test labels. Beautiful. Look at that. We've just done an optimization loop, something we spent a whole hour on before, maybe even longer, in about ten lines of code.
11:44:00
And of course, we could shorten this by making these a function. But we're going to see that later on. I'd rather us give a little bit of practice while this is still a bit fresh. Print out what's happening. Let's print out what's happening. What should we do? So because we're training for a thousand epochs, I like the idea of printing out something every 100 epochs.
11:44:21
That should be about enough of a step. Epoch. What do we got? We'll put in the epoch here with the F string and then we'll go to loss, which will be loss. And maybe we'll get the first five of those five decimal places that is. We don't have an accuracy, do we? Because we're working with regression. And we'll get the test loss out here. And that's going to be.5F as well.
11:44:50
Beautiful. Have we got any mistakes? I don't think we do. We didn't even run this code cell before. We'll just run these three again, see if we got... Look at that. Oh my goodness. Is our loss... Our loss is going down. So that means our model must be learning something. Now, what if we adjusted the learning rate here? I think if we went 0.01 or something, will that do anything? Oh, yes. Look how low our loss gets on the test data set. But let's confirm that. We've got to make some predictions.
11:45:25
Well, maybe we should do that in the next video. Yeah, this one's getting too long. But how good's that? We created a straight line data set and we've created a model to fit it. We set up a loss and an optimizer already. And we put the data on the target device. We trained and we tested so our model must be learning something. But I'd like you to give a shot at confirming that by using our plot predictions function.
11:45:48
So make some predictions with our trained model. Don't forget to turn on inference mode. And we should see some red dots here fairly close to the green dots on the next plot. Give that a shot and I'll see you in the next video. Welcome back. In the last video, we did something very exciting. We solved a smaller problem that's giving us a hint towards our larger problem. So we know that the model that we've previously been building, model two, has the capacity to learn something.
11:46:17
Now, how did we know that? Well, it's because we created this straight line data set. We replicated the architecture that we used for model one. Recall that model one didn't work very well on our classification data. But with a little bit of an adjustment such as changing the number of in features. And not too much different training code except for a different loss function because, well, we use MAE loss with regression data.
11:46:44
And we changed the learning rate slightly because we found that maybe our model could learn a bit better. And again, I'd encourage you to play around with different values of the learning rate. In fact, anything that we've changed, try and change it yourself and just see what happens. That's one of the best ways to learn what goes on with machine learning models. But we trained for the same number of epochs. We set up device agnostic code. We did a training and testing loop.
11:47:10
Look at this looks. Oh, my goodness. Well done. And our loss went down. So, hmm. What does that tell us? Well, it tells us that model two or the specific architecture has some capacity to learn something. So we must be missing something. And we're going to get to that in a minute, I promise you. But we're just going to confirm that our model has learned something and it's not just numbers on a page going down by getting visual.
11:47:35
So turn on. We're going to make some predictions and plot them. And you may have already done this because I issued that challenge at the last of at the end of the last video. So turn on evaluation mode. Let's go model two dot eval. And let's make predictions, which are also known as inference. And we're going to go with torch dot inference mode inference mode with torch dot inference mode.
11:48:02
Make some predictions. We're going to save them as why preds and we're going to use model two and we're going to pass it through ex test regression. This should all work because we've set up device agnostic code, plot data and predictions. To do this, we can of course use our plot predictions function that we imported via our helper functions dot pi function. The code for that is just a few cells above if you'd like to check that out.
11:48:27
But let's set up the train data here. Train data parameter, which is x train regression. And my goodness. Google collab. I'm already typing fast enough. You don't have to slow me down by giving me the wrong auto corrects train label equals y train regression. And then we're going to pass in our test data equals ex test regression. And then we're going to pass in test labels, which is why test regression got too many variables going on here. My goodness gracious.
11:49:03
We could have done better with naming, but this will do for now is why preds. And then if we plot this, what does it look like? Oh, no, we got an error. Now secretly, I kind of knew that that was coming ahead of time. That's the advantage of being the host of this machine learning cooking show. So type error. How do we fix this?
11:49:23
Remember how I asked you in one of the last videos what would happen if our data wasn't on the same device as our model? Well, we get an error, right? But this is a little bit different as well. We've seen this one before. We've got CUDA device type tensa to NumPy. Where is this coming from? Well, because our plot predictions function uses mapplotlib. And behind the scenes, mapplotlib references NumPy, which is another numerical computing library. However, NumPy uses a CPU rather than the GPU.
11:49:55
So we have to call dot CPU, this helpful message is telling us, call tensa dot CPU before we use our tensors with NumPy. So let's just call dot CPU on all of our tensor inputs here and see if this solves our problem. Wonderful. Looks like it does. Oh my goodness. Look at those red dots so close. Well, okay. So this just confirms our suspicions. What we kind of already knew is that our model did have some capacity to learn.
11:50:22
It's just the data set when we changed the data set it worked. So, hmm. Is it our data that our model can't learn on? Like this circular data, or is the model itself? Remember, our model is only comprised of linear functions. What is linear? Linear is a straight line, but is our data made of just straight lines?
11:50:44
I think it's got some nonlinearities in there. So the big secret I've been holding back will reveal itself starting from the next video. So if you want a head start of it, I'd go to torch and end. And if we have a look at the documentation, we've been speaking a lot about linear functions. What are these nonlinear activations? And I'll give you another spoiler. We've actually seen one of these nonlinear activations throughout this notebook.
11:51:15
So go and check that out. See what you can infer from that. And I'll see you in the next video. Let's get started with nonlinearities. Welcome back. In the last video, we saw that the model that we've been building has some potential to learn. I mean, look at these predictions. You could get a little bit better, of course, get the red dots on top of the green dots.
11:51:38
But we're just going to leave that the trend is what we're after. Our model has some capacity to learn, except this is straight line data. And we've been hinting at it a fair bit is that we're using linear functions. And if we look up linear data, what does it look like? Well, it has a quite a straight line. If we go linear and just search linear, what does this give us? Linear means straight. There we go, straight.
11:52:05
And then what happens if we search for nonlinear? I kind of hinted at this as well. Nonlinear. Oh, we get some curves. We get curved lines. So linear functions. Straight. Nonlinear functions. Hmm. Now, this is one of the beautiful things about machine learning. And I'm not sure about you, but when I was in high school, I kind of learned a concept called line of best fit, or y equals mx plus c, or
11:52:34
y equals mx plus b. And it looks something like this. And then if you wanted to go over these, you use quadratic functions and a whole bunch of other stuff. But one of the most fundamental things about machine learning is that we build neural networks and deep down neural networks are just a combination. It could be a large combination of linear functions and nonlinear functions.
11:52:56
So that's why in torch.nn, we have nonlinear activations and we have all these other different types of layers. But essentially, what they're doing deep down is combining straight lines with, if we go back up to our data, non straight lines. So, of course, our model didn't work before because we've only given it the power to use linear lines. We've only given it the power to use straight lines.
11:53:21
But our data is what? It's curved. Although it's simple, we need nonlinearity to be able to model this data set. And now, let's say we were building a pizza detection model. So let's look up some images of pizza, one of my favorite foods, images. Pizza, right? So could you model pizza with just straight lines?
11:53:43
You're thinking, Daniel, you can't be serious. A computer vision model doesn't look for just straight lines in this. And I'd argue that, yes, it does, except we also add some curved lines in here. That's the beauty of machine learning. Could you imagine trying to write the rules of an algorithm to detect that this is a pizza? Maybe you could put in, oh, it's a curve here.
11:54:02
And if you see red, no, no, no, no. Imagine if you're trying to do a hundred different foods. Your program would get really large. Instead, we give our machine learning models, if we come down to the model that we created. We give our deep learning models the capacity to use linear and nonlinear functions. We haven't seen any nonlinear layers just yet. Or maybe we've hinted at some, but that's all right. So we stack these on top of each other, these layers.
11:54:27
And then the model figures out what patterns in the data it should use, what lines it should draw to draw patterns to not only pizza, but another food such as sushi. If we wanted to build a food image classification model, it would do this. The principle remains the same. So the question I'm going to pose to you, we'll get out of this, is, we'll come down here. We've unlocked the missing piece or about to. We're going to cover it over the next couple of videos, the missing piece of our model.
11:54:58
And this is a big one. This is going to follow you out throughout all of machine learning and deep learning, nonlinearity. So the question here is, what patterns could you draw if you were given an infinite amount of straight and non straight lines? Or in machine learning terms, an infinite amount, but really it is finite. By infinite in machine learning terms, this is a technicality. It could be a million parameters. It could be as we've got probably a hundred parameters in our model.
11:55:45
So just imagine a large amount of straight and non straight lines, an infinite amount of linear and nonlinear functions. You could draw some pretty intricate patterns, couldn't you? And that's what gives machine learning and especially neural networks the capacity to not only fit a straight line here, but to separate two different circles. But also to do crazy things like drive a self-driving car, or at least power the vision system of a self-driving car.
11:56:19
Of course, after that, you need some programming to plan what to actually do with what you see in an image. But we're getting ahead of ourselves here. Let's now start diving into nonlinearity. And the whole idea here is combining the power of linear and nonlinear functions. Straight lines and non straight lines. Our classification data is not comprised of just straight lines. It's circles, so we need nonlinearity here.
11:56:44
So recreating nonlinear data, red and blue circles. We don't need to recreate this, but we're going to do it anyway for completeness. So let's get a little bit of a practice. Make and plot data. This is so that you can practice the use of nonlinearity on your own. And that plot little bit dot pie plot as PLT. We're going to go a bit faster here because we've covered this code above. So import make circles. We're just going to recreate the exact same circle data set that we've created above.
11:57:15
Number of samples. We'll create a thousand. And we're going to create x and y equals what? Make circles. Pass it in number of samples. Beautiful. Colab, please. I wonder if I can turn off autocorrect and colab. I'm happy to just see all of my errors in the flesh. See? Look at that. I don't want that. I want noise like that. Maybe I'll do that in the next video. We're not going to spend time here looking around how to do it.
11:57:40
We can work that out on the fly later. For now, I'm too excited to share with you the power of nonlinearity. So here, x, we're just going to plot what's going on. We've got two x features and we're going to color it with the flavor of y because we're doing a binary classification. And we're going to use one of my favorite C maps, which is color map. And we're going to go PLT dot CM for C map and red blue. What do we get?
11:58:10
Okay, red circle, blue circle. Hey, is it the same color as what's above? I like this color better. Did we get that right up here? Oh, my goodness. Look how much code we've written. Yeah, I like the other blue. I'm going to bring this down here. It's all about aesthetics and machine learning. It's not just numbers on a page, don't you? How could you be so crass? Let's go there. Okay, that's better color red and blue. That's small lively, isn't it? So now let's convert to train and test.
11:58:43
And then we can start to build a model with nonlinearity. Oh, this is so good. Okay, convert data to tenses and then to train and test splits. Nothing we haven't covered here before. So import torch, but it never hurts to practice code, right? Import torch from sklearn dot model selection. Import train test split so that we can split our red and blue dots randomly. And we're going to turn data into tenses.
11:59:18
And we'll go X equals torch from NumPy and we'll pass in X here. And then we'll change it into type torch dot float. Why do we do this? Well, because, oh, my goodness, autocorrect. It's getting the best of me here. You know, watching me live code this stuff and battle with autocorrect. That's what this whole course is. And we're really teaching pie torch. Am I just battling with Google collab's autocorrect?
11:59:44
We are turning it into torch dot float with a type here because why NumPy's default, which is what makes circles users behind the scenes. NumPy is actually using a lot of other machine learning libraries, pandas, built on NumPy, scikit learn, does a lot of NumPy. Matplotlib, NumPy. That's just showing there. What's the word? Is it ubiquitous, ubiquity? I'm not sure, maybe. If not, you can correct me. The ubiquity of NumPy.
12:00:10
And test sets, but we're using pie torch to leverage the power of autograd, which is what powers our gradient descent. And the fact that it can use GPUs. So we're creating training test splits here with train test split X Y. And we're going to go test size equals 0.2. And we're going to set random. Random state equals 42. And then we'll view our first five samples. Are these going to be? Tenses. Fingers crossed. We haven't got an error. Beautiful. We have tenses here.
12:00:52
Okay. Now we're up to the exciting part. We've got our data set back. I think it's time to build a model with nonlinearity. So if you'd like to peek ahead, check out TorchNN again. This is a little bit of a spoiler. Go into the nonlinear activation. See if you can find the one that we've already used. That's your challenge. Can you find the one we've already used? And go into here and search what is our nonlinear function. So give that a go and see what comes up. I'll see you in the next video.
12:01:28
Welcome back. Now put your hand up if you're ready to learn about nonlinearity. And I know I can't see your hands up, but I better see some hands up or I better feel some hands up because my hands up because nonlinearity is a magic piece of the puzzle that we're about to learn about. So let's title this section building a model with nonlinearity. So just to re-emphasize linear equals straight lines and in turn nonlinear equals non-straight lines.
12:02:03
And I left off the end of the last video, giving you the challenge of checking out the TorchNN module, looking for the nonlinear function that we've already used. Now where would you go to find such a thing and oh, what do we have here? Nonlinear activations. And there's going to be a fair few things here, but essentially all of the modules within TorchNN are either some form of layer in a neural network if we recall. Let's go to a neural network. We've seen the anatomy of a neural network.
12:02:35
Generally you'll have an input layer and then multiple hidden layers and some form of output layer. Well, these multiple hidden layers can be almost any combination of what's in TorchNN. And in fact, they can almost be any combination of function you could imagine. Whether they work or not is another question. But PyTorch implements some of the most common layers that you would have as hidden layers. And they might be pooling layers, padding layers, activation functions.
12:03:07
And they all have the same premise. They perform some sort of mathematical operation on an input. And so if we look into the nonlinear activation functions, you might have find an n dot sigmoid. Where have we used this before? There's a sigmoid activation function in math terminology. It takes some input x, performs this operation on it. And here's what it looks like if we did it on a straight line, but I think we should put this in practice.
12:03:36
And if you want an example, well, there's an example there. All of the other nonlinear activations have examples as well. But I'll let you go through all of these in your own time. Otherwise we're going to be here forever. And then dot relu is another common function. We saw that when we looked at the architecture of a classification network. So with that being said, how about we start to code a classification model with nonlinearity.
12:04:02
And of course, if you wanted to, you could look up what is a nonlinear function. If you wanted to learn more, nonlinear means the graph is not a straight line. Oh, beautiful. So that's how I'd learn about nonlinear functions. But while we're here together, how about we write some code. So let's go build a model with nonlinear activation functions. And just one more thing before, just to re-emphasize what we're doing here.
12:04:33
Before we write this code, I've got, I just remembered, I've got a nice slide, which is the question we posed in the previous video, the missing piece, nonlinearity. But the question I want you to think about is what could you draw if you had an unlimited amount of straight, in other words, linear, and non-straight, nonlinear line. So we've seen previously that we can build a model, a linear model to fit some data that's in a straight line, linear data.
12:05:02
But when we're working with nonlinear data, well, we need the power of nonlinear functions. So this is circular data. And now, this is only a 2D plot, keep in mind there. Whereas neural networks and machine learning models can work with numbers that are in hundreds of dimensions, impossible for us humans to visualize, but since computers love numbers, it's a piece of cake to them. So from torch, import, and then we're going to create our first neural network with nonlinear activations.
12:05:32
This is so exciting. So let's create a class here. We'll create circle model. We've got circle model V1 already. We're going to create circle model V2. And we'll inherit from an end dot module. And then we'll write the constructor, which is the init function, and we'll pass in self here. And then we'll go self, or super sorry, too many S words. Dot underscore underscore init, underscore. There we go. So we've got the constructor here.
12:06:04
And now let's create a layer one, self dot layer one equals just the same as what we've used before. And then dot linear. We're going to create this quite similar to the model that we've built before, except with one added feature. And we're going to create in features, which is akin to the number of X features that we have here. Again, if this was different, if we had three X features, we might change this to three.
12:06:31
But because we're working with two, we'll leave it as that. We'll keep out features as 10, so that we have 10 hidden units. And then we'll go layer two, and then dot linear. Again, these values here are very customizable because why, because they're hyper parameters. So let's line up the out features of layer two, and we'll do the same with layer three. Because layer three is going to take the outputs of layer two. So it needs in features of 10.
12:06:59
And we want layer three to be the output layer, and we want one number as output, so we'll set one here. Now, here's the fun part. We're going to introduce a nonlinear function. We're going to introduce the relu function. Now, we've seen sigmoid. Relu is another very common one. It's actually quite simple. But let's write it out first, and then dot relu. So remember, torch dot nn stores a lot of existing nonlinear activation functions, so that we don't necessarily have to code them ourselves.
12:07:32
However, if we did want to code a relu function, let me show you. It's actually quite simple. If we dive into nn dot relu, or relu, however you want to say it, I usually say relu, applies the rectified linear unit function element wise. So that means element wise on every element in our input tensor. And so it stands for rectified linear unit, and here's what it does. Basically, it takes an input.
12:07:58
If the input is negative, it turns the input to zero, and it leaves the positive inputs how they are. And so this line is not straight. Now, you could argue, yeah, well, it's straight here and then straight there, but this is a form of a nonlinear activation function. So it goes boom, if it was linear, it would just stay straight there like that. But let's see it in practice. Do you think this is going to improve our model?
12:08:23
Well, let's find out together, hey, forward, we need to implement the forward method. And here's what we're going to do. Where should we put our nonlinear activation functions? So I'm just going to put a node here. Relu is a nonlinear activation function. And remember, wherever I say function, it's just performing some sort of operation on a numerical input. So we're going to put a nonlinear activation function in between each of our layers.
12:09:01
So let me show you what this looks like, self dot layer three. We're going to start from the outside in self dot relu, and then we're going to go self dot layer two. And then we're going to go self dot relu. And then there's a fair bit going on here, but nothing we can't handle layer one. And then here's the X. So what happens is our data goes into layer one, performs a linear operation with an end up linear. Then we pass the output of layer one to a relu function.
12:09:34
So we, where's relu up here, we turn all of the negative outputs of our model of our of layer one to zero, and we keep the positives how they are. And then we do the same here with layer two. And then finally, the outputs of layer three stay as they are. We've got out features there. We don't have a relu on the end here, because we're going to pass the outputs to the sigmoid function later on. And if we really wanted to, we could put self dot sigmoid here equals an end dot sigmoid.
12:10:04
But I'm going to, that's just one way of constructing it. We're just going to apply the sigmoid function to the logits of our model, because what are the logits, the raw output of our model. And so let's instantiate our model. This is going to be called model three, which is a little bit confusing, but we're up to model three, which is circle model V two, and we're going to send that to the target device. And then let's check model three. What does this look like?
12:10:35
Wonderful. So it doesn't actually show us where the relu's appear, but it just shows us what are the parameters of our circle model V two. Now, I'd like you to have a think about this. And my challenge to you is to go ahead and see if this model is capable of working on our data, on our circular data. So we've got the data sets ready. You need to set up some training code. My challenge to you is write that training code and see if this model works.
12:11:06
But we're going to go through that over the next few videos. And also, my other challenge to you is to go to the TensorFlow Playground and recreate our neural network here. You can have two hidden layers. Does this go to 10? Well, it only goes to eight. We'll keep this at five. So build something like this. So we've got two layers with five. It's a little bit different to ours because we've got two layers with 10.
12:11:29
And then put the learning rate to 0.01. What do we have? 0.1 with stochastic gradient descent. We've been using 0.1, so we'll leave that. So this is the TensorFlow Playground. And then change the activation here. Instead of linear, which we've used before, change it to relu, which is what we're using. And press play here and see what happens. I'll see you in the next video. Welcome back. In the last video, I left off leaving the challenge of recreating this model here.
12:12:00
It's not too difficult to do. We've got two hidden layers and five neurons. We've got our data set, which looks kind of like ours. But the main points here are have to learning rate of 0.1, which is what we've been using. But to change it from, we've previously used a linear activation to change it from linear to relu, which is what we've got set up here in the code. Now, remember, relu is a popular and effective nonlinear activation function.
12:12:25
And we've been discussing that we need nonlinearity to model nonlinear data. And so that's the crux to what neural networks are. Artificial neural networks, not to get confused with the brain neural networks, but who knows? This might be how they work, too. I don't know. I'm not a neurosurgeon or a neuroscientist. Artificial neural networks are a large combination of linear. So this is straight and non-straight nonlinear functions, which are potentially able to find patterns in data.
12:13:05
And so for our data set, it's quite small. It's just a blue and a red circle. But this same principle applies for larger data sets and larger models combined linear and nonlinear functions. So we've got a few tabs going on here. Let's get rid of some. Let's come back to here. Did you try this out? Does it work? Do you think it'll work? I don't know. Let's find out together. Ready? Three, two, one. Look at that.
12:13:31
Almost instantly the training loss goes down to zero and the test loss is basically zero as well. Look at that. That's amazing. We can stop that there. And if we change the learning rate, maybe a little lower, let's see what happens. It takes a little bit longer to get to where it wants to go to. See, that's the power of changing the learning rate. Let's make it really small. What happens here? So that was about 300 epochs. The loss started to go down.
12:13:57
If we change it to be really small, oh, we're getting a little bit of a trend. Is it starting to go down? We're already surpassed the epochs that we had. So see how the learning rate is much smaller? That means our model is learning much slower. So this is just a beautiful visual way of demonstrating different values of the learning rate. We could sit here all day and that might not get to lower, but let's increase it by 10x. And that was over 1,000 epochs and it's still at about 0.5, let's say.
12:14:27
Oh, we got a better. Oh, we're going faster already. So not even at 500 or so epochs, we're about 0.4. That's the power of the learning rate. We'll increase it by another 10x. We'll reset. Start again. Oh, would you look at that much faster this time. That is beautiful. Oh, there's nothing better than watching a loss curve go down. In the world of machine learning, that is. And then we reset that again. And let's change it right back to what we had. And we get to 0 in basically under 100 epochs.
12:15:01
So that's the power of the learning rate, little visual representation. Working on learning rates, it's time for us to build an optimizer and a loss function. So that's right here. We've got our nonlinear model set up loss and optimizer. You might have already done this because the code, this is code that we've written before, but we're going to redo it for completeness and practice. So we want a loss function. We're working with logits here and we're working with binary cross entropy.
12:15:28
So what loss do we use? Binary cross entropy. Sorry, we're working with a binary classification problem. Blue dots or red dots, torch dot opt in. What are some other binary classification problems that you can think of? We want model three dot parameters. They're the parameters that we want to optimize this model here. And we're going to set our LR to 0.1, just like we had in the TensorFlow playground. Beautiful. So some other binary classification problems I can think of would be email.
12:16:03
Spam or not spam credit cards. So equals fraud or not fraud. What else? You might have insurance claims. Equals who's at fault or not at fault. If someone puts in a claim speaking about a car crash, whose fault was it? Was the person submitting the claim? Were they at fault? Or was the person who was also mentioned in the claim? Are they not at fault? So there's many more, but they're just some I can think of up the top of my head. But now let's train our model with nonlinearity.
12:16:38
Oh, we're on a roll here. Training a model with nonlinearity. So we've seen that if we introduce a nonlinear activation function within a model, remember this is a linear activation function, and if we train this, the loss doesn't go down. But if we just adjust this to add a relu in here, we get the loss going down. So hopefully this replicates with our pure PyTorch code. So let's do it, hey? So we're going to create random seeds.
12:17:12
Because we're working with CUDA, we'll introduce the CUDA random seed as well. Torch.manual seed. Again, don't worry too much if your numbers on your screen aren't exactly what mine are. That's due to the inherent randomness of machine learning. In fact, stochastic gradient descent stochastic again stands for random. And we're just setting up the seeds here so that they can be as close as possible. But the direction is more important.
12:17:38
So if my loss goes down, your loss should also go down on target device. And then we're going to go Xtrain. So this is setting up device agnostic code. We've done this before. But we're going to do it again for completeness. Just to practice every step of the puzzle. That's what we want to do. We want to have experience. That's what this course is. It's a momentum builder.
12:18:03
So that when you go to other repos and machine learning projects that use PyTorch, you can go, oh, does this code set device agnostic code? What problem are we working on? Is it binary or multi-class classification? So let's go loop through data. Again, we've done this before, but we're going to set up the epochs. Let's do 1000 epochs. Why not? So we can go for epoch in range epochs. What do we do here? Well, we want to train. So this is training code. We set our model model three dot train.
12:18:40
And I want you to start to think about how could we functionalize this training code? We're going to start to move towards that in a future video. So one is forward pass. We've got the logits. Why the logits? Well, because the raw output of our model without any activation functions towards the final layer. Classified as logits or called logits. And then we create y-pred as in prediction labels by rounding the output of torch dot sigmoid of the logits.
12:19:10
So this is going to take us from logits to prediction probabilities to prediction labels. And then we can go to, which is calculate the loss. That's from my unofficial pytorch song. Calculate the last. We go loss equals loss FN y logits. Because remember, we've got BCE with logits loss and takes in logits as first input. And that's going to calculate the loss between our models, logits and the y training labels. And we will go here, we'll calculate accuracy using our accuracy function.
12:19:58
And this one is a little bit backwards compared to pytorch, but we pass in the y training labels first. But it's constructed this way because it's in the same style as scikit line. Three, we go optimizer zero grad. We zero the gradients of the optimizer so that it can start from fresh. Calculating the ideal gradients every epoch. So it's going to reset every epoch, which is fine. Then we're going to perform back propagation pytorch is going to take care of that for us by calling loss backwards.
12:20:34
And then we will perform gradient descent. So step the optimizer to see how we should improve our model parameters. So optimizer dot step. Oh, and I want to show you speaking of model parameters. Let's check our model three dot state dig. So the relu activation function actually doesn't have any parameters. So you'll notice here, we've got weight, we've got bias of layer one, layer two, and a layer three. So the relu function here doesn't have any parameters to optimize. If we go nn dot relu.
12:21:09
Does it say what it implements? There we go. So it's just the maximum of zero or x. So it takes the input and takes the max of zero or x. And so when it takes the max of zero or x, if it's a negative number, zero is going to be higher than a negative number. So that's why it zeroes all of the negative inputs. And then it leaves the positive inputs how they are because the max of a positive input versus zero is the positive input.
12:21:38
So this has no parameters to optimize. That's why it's so effective because you think about it. Every parameter in our model needs some little bit of computation to adjust. And so the more parameters we add to our model, the more compute that is required. So generally, the kind of trade-off in machine learning is that, yes, more parameters have more of an ability to learn, but you need more compute. So let's go model three dot a vowel. And we're going to go with torch dot inference mode.
12:22:07
If I could spell inference, that'd be fantastic. We're going to do what? We're going to do the forward pass. So test logits equals model three on the test data. And then we're going to calculate the test pred labels by calling torch dot round on torch dot sigmoid on the test logits. And then we can calculate the test loss. How do we do that? And then we can also calculate the test accuracy. I'm just going to give myself some more space here.
12:22:43
So I can code in the middle of the screen equals accuracy function on what we're going to pass in y true equals y test. We're going to pass in y true equals y test. And then we will pass in y pred equals test pred. Beautiful. A final step here is to print out what's happening. Now, this will be very important because one, it's fun to know what your model is doing. And two, if our model does actually learn, I'd like to see the loss values go down and the accuracy values go up.
12:23:20
As I said, there's nothing much more beautiful in the world of machine learning than watching a loss function go down or a loss value go down and watching a loss curve go down. So let's print out the current epoch and then we'll print out the loss, which will just be the training loss. And we'll take that to four decimal places. And then we'll go accuracy here.
12:23:41
And this will be a and we'll take this to two decimal places and we'll put a little percentage sign there and then we'll break it up by putting in the test loss here and we'll put in the test loss. Because remember our model learns patterns on the training data set and then evaluates those patterns on the test data set.
12:24:03
So, and we'll pass in test act here and no doubt there might be an error or two within all of this code, but we're going to try and run this because we've seen this code before, but I think we're ready. We're training our first model here with non-linearities built into the model. You ready? Three, two, one, let's go. Oh, of course. Module torch CUDA has no attribute manuals are just a typo standard man you out. There we go. Have to sound that out.
12:24:38
Another one. What do we get wrong here? Oh, target size must be same as input size. Where did it mess up here? What do we get wrong? Test loss, test logits on Y test. Hmm. So these two aren't matching up. Model three X test and Y test. What's the size of? So let's do some troubleshooting on the fly. Hey, not everything always works out as you want. So length of X test, we've got a shape issue here. Remember how I said one of the most common issues in deep learning is a shape issue?
12:25:18
We've got the same shape here. Let's check test logits dot shape and Y test dot shape. We'll print this out. So 200. Oh, here's what we have to do. That's what we missed dot squeeze. Oh, see how I've been hinting at the fact that we needed to call dot squeeze. So this is where the discrepancy is. Our test logits dot shape. We've got an extra dimension here. And what are we getting here? A value error on the target size, which is a shape mismatch.
12:25:53
So we've got target size 200 must be the same input size as torch size 201. So did we squeeze this? Oh, that's why the training worked. Okay, so we've missed this. Let's just get rid of this. So we're getting rid of the extra one dimension by using squeeze, which is the one dimension here. We should have everything lined up. There we go. Okay. Look at that. Yes. Now accuracy has gone up, albeit not by too much. It's still not perfect.
12:26:24
So really we'd like this to be towards 100% lost to be lower. But I feel like we've got a better performing model. Don't you? Now that is the power of non linearity. All we did was we added in a relu layer or just two of them. Relu here, relu here. But what did we do? We gave our model the power of straight lines. Oh, straight linear of straight lines and non straight lines. So it can potentially draw a line to separate these circles.
12:26:56
So in the next video, let's draw a line, plot our model decision boundary using our function and see if it really did learn anything. I'll see you there. Welcome back. In the last video, we trained our first model, and as you can tell, I've got the biggest smile on my face, but we trained our first model that harnesses both the power of straight lines and non straight lines or linear functions and non linear functions.
12:27:22
And by the 1000th epoch, we look like we're getting a bit better results than just pure guessing, which is 50%. Because we have 500 samples of red dots and 500 samples of blue dots. So we have evenly balanced classes. Now, we've seen that if we added a relu activation function with a data set similar to ours with a TensorFlow playground, the model starts to fit. But it doesn't work with just linear. There's a few other activation functions that you could play around with here.
12:27:51
You could play around with the learning rate, regularization. If you're not sure what that is, I'll leave that as extra curriculum to look up. But we're going to retire the TensorFlow program for now because we're going to go back to writing code. So let's get out of that. Let's get out of that. We now have to evaluate our model because right now it's just numbers on a page. So let's write down here 6.4. What do we like to do to evaluate things? It's visualize, visualize, visualize.
12:28:18
So evaluating a model trained with nonlinear activation functions. And we also discussed the point that neural networks are really just a big combination of linear and nonlinear functions trying to draw patterns in data. So with that being said, let's make some predictions with our Model 3, our most recently trained model. We'll put it into a Val mode and then we'll set up inference mode. And then we'll go yprads equals torch dot round and then torch dot sigmoid.
12:28:55
We could functionalize this, of course, Model 3 and then pass in X test. And you know what? We're going to squeeze these here because we ran into some troubles in the previous video. I actually really liked that we did because then we got to troubleshoot a shape error on the fly because that's one of the most common issues you're going to come across in deep learning. So yprads, let's check them out and then let's check out y test. You want y test 10.
12:29:23
So remember, when we're evaluating predictions, we want them to be in the same format as our original labels. We want to compare apples to apples. And if we compare the format here, do these two things look the same? Yes, they do. They're both on CUDA and they're both floats. We can see that it's got this one wrong. Whereas the other ones look pretty good. Hmm, this might look pretty good if we visualize it.
12:29:47
So now let's, you might have already done this because I issued the challenge of plotting the decision boundaries. Plot decision boundaries and let's go PLT dot figure and we're going to set up the fig size to equal 12.6 because, again, one of the advantages of hosting a machine learning cooking show is that you can code ahead of time. And then we can go PLT dot title is train. And then we're going to call our plot decision boundary function, which we've seen before. Plot decision boundary.
12:30:20
And we're going to pass this one in. We could do model three, but we could also pass it in our older models to model one that doesn't use it on the reality. In fact, I reckon that'll be a great comparison. So we'll also create another plot here for the test data and this will be on index number two. So remember, subplot is a number of rows, number of columns, index where the plot appears. We'll give this one a title. Plot dot title. This will be test and Google Colab. I didn't want that.
12:30:54
As I said, this course is also a battle between me and Google Colab's autocorrect. So we're going model three and we'll pass in the test data here. And behind the scenes, our plot decision boundary function will create a beautiful graphic for us, perform some predictions on the X, the features input, and then we'll compare them with the Y values. Let's see what's going on here. Oh, look at that. Yes, our first nonlinear model.
12:31:25
Okay, it's not perfect, but it is certainly much better than the models that we had before. Look at this. Model one has no linearity. Model one equals no nonlinearity. I've got double negative there. Whereas model three equals has nonlinearity. So do you see the power of nonlinearity or better yet the power of linearity or linear straight lines with non straight lines? So I feel like we could do better than this, though. Here's your challenge is to can you improve model three to do better?
12:32:02
What did we get? 79% accuracy to do better than 80% accuracy on the test data. I think you can. So that's the challenge. And if you're looking for hints on how to do so, where can you look? Well, we've covered this improving a model. So maybe you add some more layers, maybe you add more hidden units. Maybe you fit for longer. Maybe you if you add more layers, you put a relio activation function on top of those as well. Maybe you lower the learning rate because right now we've got 0.1.
12:32:36
So give this a shot, try and improve it. I think you can do it. But we're going to push forward. That's going to be your challenge for some extra curriculum. I think in the next section, we've seen our nonlinear activation functions in action. Let's write some code to replicate them. I'll see you there. Welcome back. In the last video, I left off with the challenge of improving model three to do better than 80% accuracy on the test data. I hope you gave it a shot.
12:33:06
But here are some of the things I would have done. As I potentially add more layers, I maybe increase the number of hidden units, and then if we needed to fit for longer and maybe lower the learning rate to 0.01. But I'll leave that for you to explore because that's the motto of the data scientists, right? Is to experiment, experiment, experiment. So let's go in here. We've seen our nonlinear activation functions in practice. Let's replicate them. So replicating nonlinear activation functions.
12:33:38
And remember neural networks rather than us telling the model what to learn. We give it the tools to discover patterns in data. And it tries to figure out the best patterns on its own. And what are these tools? That's right down here. We've seen this in action. And these tools are linear and nonlinear functions. So a neural network is a big stack of linear and nonlinear functions. For us, we've only got about four layers or so, four or five layers.
12:34:21
But as I said, other networks can get much larger. But the premise remains. Some form of linear and nonlinear manipulation of the data. So let's get out of this. Let's make our workspace a little bit more cleaner. Replicating nonlinear activation functions. So let's create a tensor to start with. Everything starts from the tensor. And we'll go A equals torch A range. And we're going to create a range from negative 10 to 10 with a step of one.
12:34:52
And we can set the D type here to equal torch dot float 32. But we don't actually need to. That's going to be the default. So if we set A here, A dot D type. Then we've got torch float 32 and I'm pretty sure if we've got rid of that. Oh, we've got torch in 64. Why is that happening? Well, let's check out A. Oh, it's because we've got integers as our values because we have a step as one. If we turn this into a float, what's going to happen? We get float 32. But we'll keep it.
12:35:29
Otherwise, this is going to be what? About a hundred numbers? Yeah, no, that's too many. Let's keep it at negative 10 to 10 and we'll set the D type here to torch float 32. Beautiful. So it looks like PyTorch's default data type for integers is in 64. But we're going to work with float 32 because float 32, if our data wasn't float 32 with the functions we're about to create, we might run into some errors. So let's visualize this data.
12:35:59
I want you to guess, is this a straight line or non-straight line? You've got three seconds. One two three. Straight line. There we go. We've got negative 10 to positive 10 up here or nine. Close enough. And so how would we turn this straight line? If it's a straight line, it's linear. How would we perform the relu activation function on this? Now, we could of course call torch relu on A. Actually, let's in fact just plot this. PLT dot plot on torch relu. What does this look like?
12:36:39
Boom, there we go. But we want to replicate the relu function. So let's go nn dot relu. What does it do? We've seen this before. So we need the max. We need to return based on an input. We need the max of zero and x. So let's give it a shot. We'll come here. Again, we need more space. There can never be enough code space here. I like writing lots of code. I don't know about you. But let's go relu. We'll take an input x, which will be some form of tensor. And we'll go return torch dot maximum.
12:37:13
I think you could just do torch dot max. But we'll try maximum. Torch dot tensor zero. So the maximum is going to return the max between whatever this is. One option and whatever the other option is. So inputs must be tensors. So maybe we could just give a type hint here that this is torch dot tensor. And this should return a tensor too. Return torch dot tensor. Beautiful. You're ready to try it out. Let's see what our relu function does. Relu A. Wonderful.
12:37:52
It looks like we got quite a similar output to before. Here's our original A. So we've got negative numbers. There we go. So recall that the relu activation function turns all negative numbers into zero because it takes the maximum between zero and the input. And if the input's negative, well then zero is bigger than it. And it leaves all of the positive values as they are. So that's the beauty of relu. Quite simple, but very effective. So let's plot relu activation function. Our custom one.
12:38:26
We will go PLT dot plot. We'll call our relu function on A. Let's see what this looks like. Oh, look at us go. Well done. Just the exact same as the torch relu function. Easy as that. And what's another nonlinear activation function that we've used before? Well, I believe one of them is if we go down to here, what did we say before? Sigmoid. Where is that? Where are you, Sigmoid? Here we go. Hello Sigmoid. Oh, this has got a little bit more going on here.
12:39:02
One over one plus exponential of negative x. So Sigmoid or this little symbol for Sigmoid of x, which is an input. We get this. So let's try and replicate this. I might just bring this one in here. Right now, let's do the same for Sigmoid. So what do we have here? Well, we want to create a custom Sigmoid. And we want to have some sort of input, x. And we want to return one divided by, do we have the function in Sigmoid? One divided by one plus exponential.
12:39:43
One plus torch dot exp for exponential on negative x. And we might put the bottom side in brackets so that it does that operation. I reckon that looks all right to me. So one divided by one plus torch exponential of negative x. Do we have that? Yes, we do. Well, there's only one real way to find out. Let's plot the torch version of Sigmoid. Torch dot Sigmoid and we'll pass in x. See what happens. And then, oh, we have a. My bad. A is our tensor. What do we get? We get a curved line. Wonderful.
12:40:25
And then we go plt dot plot. And we're going to use our Sigmoid function on a. Did we replicate torch's Sigmoid function? Yes, we did. Ooh now. See, this is what's happening behind the scenes with our neural networks. Of course, you could do more complicated activation functions or layers and whatnot. And you can try to replicate them. In fact, that's a great exercise to try and do. But we've essentially across the videos and the sections that we've done, we've replicated our linear layer.
12:40:54
And we've replicated the relu. So we've actually built this model from scratch, or we could if we really wanted to. But it's a lot easier to use PyTorch's layers because we're building neural networks here like Lego bricks, stacking together these layers in some way, shape, or form. And because they're a part of PyTorch, we know that they've been error-tested and they compute as fast as possible behind the scenes and use GPU and get a whole bunch of benefits.
12:41:18
PyTorch offers a lot of benefits by using these layers rather than writing them ourselves. And so this is what our model is doing. It's literally like to learn these values and decrease the loss function and increase the accuracy. It's combining linear layers and nonlinear layers or nonlinear functions. Where's our relu function here? A relu function like this behind the scenes. So just combining linear and nonlinear functions to fit a data set.
12:41:47
And that premise remains even on our small data set and on very large data sets and very large models. So with that being said, I think it's time for us to push on. We've covered a fair bit of code here. But we've worked on a binary classification problem. Have we worked on a multi-class classification problem yet? Do we have that here? Where's my fun graphic? We have multi-class classification. I think that's what we cover next.
12:42:17
We're going to put together all of the steps in our workflow that we've covered for binary classification. But now let's move on to a multi-class classification problem. If you're with me, I'll see you in the next video. Welcome back. In the last few videos we've been harnessing the power of nonlinearity. Specifically non-straight line functions and we replicated some here. And we learned that a neural network combines linear and nonlinear functions to find patterns in data.
12:42:49
And for our simple red versus blue dots, once we added a little bit of nonlinearity, we found the secret source of to start separating our blue and red dots. And I also issued you the challenge to try and improve this and I think you can do it. So hopefully you've given that a go. But now let's keep pushing forward. We're going to reiterate over basically everything that we've done, except this time from the point of view of a multi-class classification problem.
12:43:15
So I believe we're up to section eight, putting it all together with a multi-class classification problem. Beautiful. And recall the difference between binary classification equals one thing or another such as cat versus dog. If you were building a cat versus dog image classifier, spam versus not spam for say emails that were spam or not spam or even internet posts on Facebook or Twitter or one of the other internet services. And then fraud or not fraud for credit card transactions.
12:43:57
And then multi-class classification is more than one thing or another. So we could have cat versus dog versus chicken. So I think we've got all the skills to do this. Our architecture might be a little bit different for a multi-class classification problem. But we've got so many building blocks now. It's not funny. Let's clean up this and we'll add some more code cells and just to reiterate. So we've gone over nonlinearity.
12:44:29
The question is what could you draw if you had an unlimited amount of straight linear and non-straight, nonlinear lines, I believe you could draw some pretty intricate patterns. And that is what our neural networks are doing behind the scenes. And so we also learned that if we wanted to just replicate some of these nonlinear functions, some of the ones that we've used before, we could create a range. Linear activation is just the line itself.
12:44:54
And then if we wanted to do sigmoid, we get this curl here. And then if we wanted to do relu, well, we saw how to replicate the relu function as one. These both are nonlinear. And of course, torch.nn has far more nonlinear activations where they came from just as it has far more different layers. And you'll get used to these with practice. And that's what we're doing here. So let's go back to the keynote. So this is what we're going to be working on. Multi-class classification.
12:45:27
So there's one of the big differences here. We use the softmax activation function versus sigmoid. There's another big difference here. Instead of binary cross entropy, we use just cross entropy. But I think most of it's going to stay the same. We're going to see this in action in a second. But let's just describe our problem space. Just to go visual, we've covered a fair bit here. Well done, everyone. So binary versus multi-class classification. Binary one thing or another. Zero or one.
12:46:00
Multi-class could be three things. Could be a thousand things. Could be 5,000 things. Could be 25 things. So more than one thing or another. But that's the basic premise we're going to go with. Let's create some data, hey? 8.1. Creating a 20 multi-class data set. And so to create our data set, we're going to import our dependencies. We're going to re-import torch, even though we already have it. Just for a little bit of completeness. And we're going to go map plotlib.
12:46:33
So we can plot, as always, we like to get visual where we can. Visualize visualize visualize. We're going to import from scikitlearn.datasets. Let's get make blobs. Now, where would I get this from? SKlearn.datasets. What do we get? 20 data sets. Do we have classification? 20 data sets. Do we have blobs? If we just go make scikitlearn. Classification data sets. What do we get? Here's one option. There's also make blobs. Beautiful. Make blobs. This is a code for that.
12:47:20
So let's just copy this in here. And make blobs. We're going to see this in action anyway. Make blobs. As you might have guessed, it makes some blobs for us. I like blobs. It's a fun word to say. Blobs. So we want train test split because we want to make a data set and then we want to split it into train and test. Let's set the number of hyper parameters. So set the hyper parameters for data creation. Now I got these from the documentation here. Number of samples. How many blobs do we want?
12:47:53
How many features do we want? So say, for example, we wanted two different classes. That would be binary classification. Say, for example, you wanted 10 classes. You could set this to 10. And we're going to see what the others are in practice. But if you want to read through them, you can well and truly do that. So let's set up. We want num classes. Let's double what we've been working with. We've been working with two classes, red dots or blue dots. Let's step it up a notch.
12:48:19
We'll go to four classes. Watch out, everyone. And we're going to go number of features will be two. So we have the same number of features. And then the random seed is going to be 42. You might be wondering why these are capitalized. Well, generally, if we do have some hyper parameters that we say set at the start of a notebook, you'll find it's quite common for people to write them as capital letters just to say that, hey, these are some settings that you can change.
12:48:46
You don't have to, but I'm just going to introduce that anyway because you might stumble upon it yourself. So create multi-class data. We're going to use the make blobs function here. So we're going to create some x blobs, some feature blobs and some label blobs. Let's see what these look like in a second. I know I'm just saying blobs a lot. But we pass in here, none samples. How many do we want? Let's create a thousand as well.
12:49:14
That could really be a hyper parameter, but we'll just leave that how it is for now. Number of features is going to be num features. Centres equals num classes. So we're going to create four classes because we've set up num classes equal to four. And then we're going to go center standard deviation. We'll give them a little shake up, add a little bit of randomness in here. Give the clusters a little shake up. We'll mix them up a bit. Make it a bit hard for our model.
12:49:48
But we'll see what this does in a second. Random state equals random seed, which is our favorite random seed 42. Of course, you can set it whatever number you want, but I like 42. Oh, and we need a comma here, of course. Beautiful. Now, what do we have to do here? Well, because we're using scikit-learn and scikit-learn leverages NumPy. So let's turn our data into tenses. Turn data into tenses. And how do we do that? Well, we grab x blob and we call torch from NumPy from NumPy.
12:50:22
If I could type, that would be fantastic. That's all right. We're doing pretty well today. Haven't made too many typos. We did make a few in a couple of videos before, but hey. I'm only human. So we're going to torch from NumPy and we're going to pass in the y blob. And we'll turn it into torch dot float because remember NumPy defaults as float 64, whereas PyTorch likes float 32. So split into training and test. And we're going to create x blob train y or x test. x blob test.
12:51:01
We'll keep the blob nomenclature here. y blob train and y blob test. And here's again where we're going to leverage the train test split function from scikit-learn. So thank you for that scikit-learn. x blob and we're going to pass the y blob. So features, labels, x is the features, y are the labels. And a test size, we've been using a test size of 20%. That means 80% of the data will be for the training data. That's a fair enough split with our data set.
12:51:31
And we're going to set the random seed to random seed because generally normally train test split is random, but because we want some reproducibility here, we're passing random seeds. Finally, we need to get visual. So let's plot the data. Right now we've got a whole bunch of code and a whole bunch of talking, but not too much visuals going on. So we'll write down here, visualize, visualize, visualize. And we can call in plot.figure. What size do we want?
12:52:02
I'm going to use my favorite hand in poker, which is 10-7, because it's generally worked out to be a good plot size. In my experience, anyway, we'll go x blob. And we want the zero index here, and then we'll grab x blob as well. And you might notice that we're visualizing the whole data set here. That's perfectly fine. We could visualize, train and test separately if we really wanted to, but I'll leave that as a level challenge to you. And we're going to go red, yellow, blue. Wonderful.
12:52:40
What do we get wrong? Oh, of course we got something wrong. Santa STD, did we spell center wrong? Cluster STD. That's what I missed. So, cluster STD. Standard deviation. What do we get wrong? Random seed. Oh, this needs to be random state. Oh, another typo. You know what? Just as I said, I wasn't getting too many typos. I'll get three. There we go. Look at that. Our first multi-class classification data set. So if we set this to zero, what does it do to our clusters?
12:53:11
Let's take note of what's going on here, particularly the space between all of the dots. Now, if we set this cluster STD to zero, what happens? We get dots that are really just, look at that. That's too easy. Let's mix it up, all right? Now, you can pick whatever value you want here. I'm going to use 1.5, because now we need to build a model that's going to draw some lines between these four colors. Two axes, four different classes.
12:53:36
But it's not going to be perfect because we've got some red dots that are basically in the blue dots. And so, what's our next step? Well, we've got some data ready. It's now time to build a model. So, I'll see you in the next video. Let's build our first multi-class classification model.
12:53:57
Welcome back. In the last video, we created our multi-class classification data set, using scikit-learn's make-blobs function. And now, why are we doing this? Well, because we're going to put all of what we've covered so far together. But instead of using binary classification or working with binary classification data, we're going to do it with multi-class classification data. So, with that being said, let's get into building our multi-class classification model.
12:54:26
So, we'll create a little heading here. Building a multi-class classification model in PyTorch. And now, I want you to have a think about this. We spent the last few videos covering non-linearity. Does this data set need non-linearity? As in, could we separate this data set with pure straight lines? Or do we need some non-straight lines as well? Have a think about that. It's okay if you're not sure, we're going to be building a model to fit this data anyway, or draw patterns in this data anyway.
12:55:03
And now, before we get into coding a model, so for multi-class classification, we've got this. For the input layer shape, we need to define the in features. So, how many in features do we have for the hidden layers? Well, we could set this to whatever we want, but we're going to keep it nice and simple for now. For the number of neurons per hidden layer, again, this could be almost whatever we want, but because we're working with a relatively small data set,
12:55:32
we've only got four different classes, we've only got a thousand data points, we'll keep it small as well, but you could change this. Remember, you can change any of these because they're hyper parameters. For the output layer shape, well, how many output features do we want? We need one per class, how many classes do we have? We have four clusters of different dots here, so we'll need four output features.
12:55:59
And then if we go back, we have an output activation of softmax, we haven't seen that yet, and then we have a loss function, rather than binary cross entropy, we have cross entropy. And then optimizer as well is the same as binary classification, two of the most common are SGDs, stochastic gradient descent, or the atom optimizer, but of course, the torch.optim package has many different options as well. So let's push forward and create our first multi-class classification model.
12:56:28
First, we're going to create, we're going to get into the habit of creating device agnostic code, and we'll set the device here, equals CUDA, nothing we haven't seen before, but again, we're doing this to put it all together, so that we have a lot of practice. Is available, else CPU, and let's go device. So we should have a GPU available, beautiful CUDA. Now, of course, if you don't, you can go change runtime type, select GPU here,
12:57:03
that will restart the runtime, you'll have to run all of the code that's before this cell as well, but I'm going to be using a GPU. You don't necessarily need one because our data set's quite small, and our models aren't going to be very large, but we set this up so we have device agnostic code. And so let's build a multi-class classification model. Look at us go, just covering all of the foundations of classification in general here,
12:57:31
and we now know that we can combine linear and non-linear functions to create neural networks that can find patterns in almost any kind of data. So I'm going to call my class here blob model, and it's going to, of course, inherit from nn.module, and we're going to upgrade our class here. We're going to take some inputs here, and I'll show you how to do this. If you're familiar with Python classes, you would have already done stuff like this, but we're going to set some parameters for our models,
12:58:01
because as you write more and more complex classes, you'll want to take inputs here. And I'm going to pre-build the, or pre-set the hidden units parameter to eight. Because I've decided, you know what, I'm going to start off with eight hidden units, and if I wanted to change this to 128, I could. But in the constructor here, we've got some options. So we have input features. We're going to set these programmatically as inputs to our class when we instantiate it.
12:58:28
The same with output features as well. And so here, we're going to call self. Oh no super. Sorry. I always get this mixed up dot init. And underscore underscore. Beautiful. So we could do a doc string here as well. So let's write in this. Initializes multi-class classification. If I could spell class e-fication model. Oh, this is great. And then we have some arcs here. This is just a standard way of writing doc strings. If you want to find out, this is Google Python doc string guide.
12:59:13
There we go. Google Python style guide. This is where I get mine from. You can scroll through this. This is just a way to write Python code. Yeah, there we go. So we've got a little sentence saying what's going on. We've got arcs. We've got returns and we've got errors if something's going on. So I highly recommend checking that out. Just a little tidbit. So this is if someone was to use our class later on. They know what the input features are.
12:59:39
Input features, which is an int, which is number of input features to the model. And then, of course, we've got output features, which is also an int. Which is number of output features of the model. And we've got the red line here is telling us we've got something wrong, but that's okay. And then the hidden features. Oh, well, this is number of output classes for the case of multi-class classification. And then the hidden units.
13:00:13
Int and then number of hidden units between layers and then the default is eight. Beautiful. And then under that, we'll just do that. Is that going to fix itself? Yeah, there we go. We could put in what it returns. Returns, whatever it returns. And then an example use case, but I'll leave that for you to fill out. If you like. So let's instantiate some things here. What we might do is write self dot linear layer stack. Self dot linear layer stack. And we will set this as nn dot sequential.
13:00:56
Ooh, we haven't seen this before. But we're just going to look at a different way of writing a model here. Previously, when we created a model, what did we do? Well, we instantiated each layer as its own parameter here. And then we called on them one by one, but we did it in a straightforward fashion. So that's why we're going to use sequential here to just step through our layers. We're not doing anything too fancy, so we'll just set up a sequential stack of layers here.
13:01:24
And recall that sequential just steps through, passes the data through each one of these layers one by one. And because we've set up the parameters up here, input features can equal to input features. And output features, what is this going to be? Is this going to be output features or is this going to be hidden units? It's going to be hidden units because it's not the final layer. We want the final layer to output our output features.
13:01:53
So input features, this will be hidden units because remember the subsequent layer needs to line up with the previous layer. Output features, we're going to create another one that outputs hidden units. And then we'll go in n.linear in features equals hidden units because it takes the output features of the previous layer. So as you see here, the output features of this feeds into here. The output features of this feeds into here. And then finally, this is going to be our final layer.
13:02:25
We'll do three layers. Output features equals output features. Wonderful. So how do we know the values of each of these? Well, let's have a look at xtrain.shape and ytrain.shape. So in the case of x, we have two input features. And in the case of y, well, this is a little confusing as well because y is a scalar. But what do you think the values for y are going to be? Well, let's go NP. Or is there torch.unique? I'm not sure. Let's find out together, hey? Torch unique.
13:03:03
Zero on one, ytrain. Oh, we need y blob train. That's right, blob. I'm too used to writing blob. And we need blob train, but I believe it's the same here. And then blob. There we go. So we have four classes. So we need an output features value of four. And now if we wanted to add nonlinearity here, we could put it in between our layers here like this. But I asked the question before, do you think that this data set needs nonlinearity? Well, let's leave it in there to begin with.
13:03:43
And one of the challenges for you, oh, do we need commerce here? I think we need commerce here. One of the challenges for you will be to test the model with nonlinearity and without nonlinearity. So let's just leave it in there for the time being. What's missing from this? Well, we need a forward method. So def forward self X. What can we do here? Well, because we've created this as a linear layer stack using nn.sequential, we can just go return linear layer stack and pass it X.
13:04:18
And what's going to happen? Whatever input goes into the forward method is just going to go through these layers sequentially. Oh, we need to put self here because we've initialized it in the constructor. Beautiful. And now let's create an instance of blob model and send it to the target device. We'll go model four equals blob model. And then we can use our input features parameter, which is this one here. And we're going to pass it a value of what? Two.
13:04:55
And then output features. Why? Because we have two X features. Now, the output feature is going to be the same as the number of classes that we have for. If we had 10 classes, we'd set it to 10. So we'll go four. And then the hidden units is going to be eight by default. So we don't have to put this here, but we're going to put it there anyway. And then, of course, we're going to send this to device. And then we're going to go model four. What do we get wrong here?
13:05:26
Unexpected keyword argument output features. Do we spell something wrong? No doubt. We've got a spelling mistake. Output features. Output features. Oh, out features. Ah, that's what we needed. Out features, not output. I've got a little confused there. Okay. There we go. Okay, beautiful. So just recall that the parameter here for an end up linear. Did you pick up on that? Is out features not output features.
13:05:59
Output features, a little confusing here, is our final layout output layers number of features there. So we've now got a multi-class classification model that lines up with the data that we're using. So the shapes line up. Beautiful. Well, what's next? Well, we have to create a loss function. And, of course, a training loop. So I'll see you in the next few videos. And let's do that together. Welcome back. In the last video, we created our multi-class classification model.
13:06:31
And we did so by subclassing an end up module. And we set up a few parameters for our class constructor here. So that when we made an instance of the blob model, we could customize the input features. The output features. Remember, this lines up with how many features X has. And the output features here lines up with how many classes are in our data. So if we had 10 classes, we could change this to 10. And it would line up. And then if we wanted 128 hidden units, well, we could change that.
13:07:02
So we're getting a little bit more programmatic with how we create models here. And as you'll see later on, a lot of the things that we've built in here can also be functionalized in a similar matter. But let's keep pushing forward. What's our next step? If we build a model, if we refer to the workflow, you'd see that we have to create a loss function. And an optimizer for a multi-class classification model. And so what's our option here for creating a loss function?
13:07:36
Where do we find loss functions in PyTorch? I'm just going to get out of this. And I'll make a new tab here. And if we search torch.nn Because torch.nn is the basic building box for graphs. In other words, neural networks. Where do we find loss functions? Hmm, here we go. Beautiful. So we've seen that L1 loss or MSE loss could be used for regression, predicting a number. And I'm here to tell you as well that for classification, we're going to be looking at cross entropy loss.
13:08:07
Now, this is for multi-class classification. For binary classification, we work with BCE loss. And of course, there's a few more here, but I'm going to leave that as something that you can explore on your own. Let's jump in to cross entropy loss. So what do we have here? This criterion computes. Remember, a loss function in PyTorch is also referred to as a criterion. You might also see loss function referred to as cost function, C-O-S-T.
13:08:36
But I call them loss functions. So this criterion computes the cross entropy loss between input and target. Okay, so the input is something, and the target is our target labels. It is useful when training a classification problem with C classes. There we go. So that's what we're doing. We're training a classification problem with C classes, C is a number of classes. If provided the optional argument, weight should be a 1D tensor assigning a weight to each of the classes.
13:09:06
So we don't have to apply a weight here, but why would you apply a weight? Well, it says, if we look at weight here, this is particularly useful when you have an unbalanced training set. So just keep this in mind as you're going forward. If you wanted to train a dataset that has imbalanced samples, in our case we have the same number of samples for each class, but sometimes you might come across a dataset with maybe you only have 10 yellow dots.
13:09:33
And maybe you have 500 blue dots and only 100 red and 100 light blue dots. So you have an unbalanced dataset. So that's where you can come in and have a look at the weight parameter here. But for now, we're just going to keep things simple. We have a balanced dataset, and we're going to focus on using this loss function. If you'd like to read more, please, you can read on here. And if you wanted to find out more, you could go, what is cross entropy loss?
13:09:59
And I'm sure you'll find a whole bunch of loss functions. There we go. There's the ML cheat sheet. I love that. The ML glossary, that's one of my favorite websites. Towards data science, you'll find that website, Wikipedia. Machine learning mastery is also another fantastic website. But you can do that all in your own time. Let's code together, hey. We'll set up a loss function. Oh, and one more resource before we get into code is that we've got the architecture,
13:10:26
well, the typical architecture of a classification model. The loss function for multi-class classification is cross entropy or torch.nn.cross entropy loss. Let's code it out. If in doubt, code it out. So create a loss function for multi-class classification. And then we go, loss fn equals, and then dot cross entropy loss. Beautiful. And then we want to create an optimizer. Create an optimizer for multi-class classification. And then the beautiful thing about optimizers is they're quite flexible.
13:11:12
They can go across a wide range of different problems. So the optimizer. So two of the most common, and I say most common because they work quite well. Across a wide range of problems. So that's why I've only listed two here. But of course, within the torch dot opt in module, you will find a lot more different optimizers. But let's stick with SGD for now. And we'll go back and go optimizer equals torch dot opt in for optimizer SGD for stochastic gradient descent.
13:11:43
The parameters we want our optimizer to optimize model four, we're up to our fourth model already. Oh my goodness. Model four dot parameters. And we'll set the learning rate to 0.1. Of course, you could change the learning rate if you wanted to. In fact, I'd encourage you to see what happens if you do because why the learning rate is a hyper parameter.
13:12:09
I'm better at writing code than I am at spelling. You can change. Wonderful. So we've now got a loss function and an optimizer for a multi class classification problem. What's next? Well, we could start to build. Building a training loop. We could start to do that, but I think we have a look at what the outputs of our model are. So more specifically, so getting prediction probabilities for a multi class pie torch model.
13:12:47
So my challenge to you before the next video is to have a look at what happens if you pass x blob test through a model. And remember, what is a model's raw output? What is that referred to as? Oh, I'll let you have a think about that before the next video. I'll see you there. Welcome back. In the last video, we created a loss function and an optimizer for our multi class classification model. And recall the loss function measures how wrong our model's predictions are.
13:13:21
And the optimizer optimizer updates our model parameters to try and reduce the loss. So that's what that does. And I also issued the challenge of doing a forward pass with model four, which is the most recent model that we created. And oh, did I just give you some code that wouldn't work? Did I do that on purpose? Maybe, maybe not, you'll never know. So if this did work, what are the raw outputs of our model? Let's get some raw outputs of our model.
13:14:00
And if you recall, the raw outputs of a model are called logits. So we got a runtime error expected. All tensors to be on the same device are of course. Why did this come up? Well, because if we go next model for dot parameters, and if we check device, what happens here? Oh, we need to bring this in. Our model is on the CUDA device, whereas our data is on the CPU still. Can we go X? Is our data a tensor? Can we check the device parameter of that? I think we can.
13:14:35
I might be proven wrong here. Oh, it's on the CPU. Of course, we're getting a runtime error. Did you catch that one? If you did, well done. So let's see what happens. But before we do a forward pass, how about we turn our model into a vowel mode to make some predictions with torch dot inference mode? We'll make some predictions. We don't necessarily have to do this because it's just tests, but it's a good habit.
13:14:59
Oh, why prads? Equals, what do we get? Why prads? And maybe we'll just view the first 10. What do we get here? Oh, my goodness. How much are numbers on a page? Is this the same format as our data or our test labels? Let's have a look. No, it's not. Okay. Oh, we need why blob test. Excuse me. We're going to make that mistake a fair few times here. So we need to get this into the format of this. Hmm.
13:15:33
How can we do that? Now, I want you to notice one thing as well is that we have one value here per one value, except that this is actually four values. Now, why is that? We have one, two, three, four. Well, that is because we set the out features up here. Our model outputs four features per sample. So each sample right now has four numbers associated with it. And what are these called? These are the logits.
13:16:01
Now, what we have to do here, so let's just write this down in order to evaluate and train and test our model. We need to convert our model's outputs, outputs which are logits to prediction probabilities, and then to prediction labels. So we've done this before, but for binary classification. So we have to go from logits to predprobs to pred labels. All right, I think we can do this. So we've got some logits here. Now, how do we convert these logits to prediction probabilities?
13:16:43
Well, we use an activation function. And if we go back to our architecture, what's our output activation here? For a binary classification, we use sigmoid. But for multi-class classification, these are the two main differences between multi-class classification and binary classification. One uses softmax, one uses cross entropy. And it's going to take a little bit of practice to know this off by heart.
13:17:06
It took me a while, but that's why we have nice tables like this. And that's why we write a lot of code together. So we're going to use a softmax function here to convert out logits. Our models raw outputs, which is this here, to prediction probabilities. And let's see that. So convert our models, logit outputs to prediction probabilities. So let's create why predprobs. So I like to call prediction probabilities predprobs for short.
13:17:39
So torch dot softmax. And then we go why logits. And we want it across the first dimension. So let's have a look. If we print why logits, we'll get the first five values there. And then look at the conversion here. Why logits? Oh, why predprobs? That's what we want to compare. Predprobs. Five. Let's check this out. Oh, what did we get wrong here? Why logits? Do we have why logits? Oh, no. We should change this to why logits, because really that's the raw output of our model here.
13:18:16
Why logits? Let's rerun that. Check that. We know that these are different to these, but we ideally want these to be in the same format as these, our test labels. These are our models predictions. And now we should be able to convert. There we go. Okay, beautiful. What's happening here? Let's just get out of this. And we will add a few code cells here. So we have some space. Now, if you wanted to find out what's happening with torch dot softmax, what could you do?
13:18:45
We could go torch softmax. See what's happening. Softmax. Okay, so here's the function that's happening. We replicated some nonlinear activation functions before. So if you wanted to replicate this, what could you do? Well, if in doubt, code it out. You could code this out. You've got the tools to do so. We've got softmax to some X input takes the exponential of X. So torch exponential over the sum of torch exponential of X. So I think you could code that out if you wanted to.
13:19:16
But let's for now just stick with what we've got. We've got some logits here, and we've got some softmax, some logits that have been passed through the softmax function. So that's what's happened here. We've passed our logits as the input here, and it's gone through this activation function. These are prediction probabilities. And you might be like, Daniel, these are still just numbers on a page. But you also notice that none of them are negative.
13:19:43
Okay, and there's another little tidbit about what's going on here. If we sum one of them up, let's get the first one. Will this work? And if we go torch dot sum, what happens? Ooh, they all sum up to one. So that's one of the effects of the softmax function. And then if we go torch dot max of Y-pred probes. So this is a prediction probability.
13:20:12
For multi class, you'll find that for this particular sample here, the 0th sample, this is the maximum number. And so our model, what this is saying is our model is saying, this is the prediction probability. This is how much I think it is class 0. This number here, it's in order. This is how much I think it is class 1. This is how much I think it is class 2.
13:20:34
This is how much I think it is class 3. And so we have one value for each of our four classes, a little bit confusing because it's 0th indexed. But the maximum value here is this index. And so how would we get the particular index value of whatever the maximum number is across these values? Well, we can take the argmax and we get tensor 1. So for this particular sample, this one here, our model, and these guesses or these predictions aren't very good.
13:21:08
Why is that? Well, because our model is still just predicting with random numbers, we haven't trained it yet. So this is just random output here, basically. But for now, the premise still remains that our model thinks that for this sample using random numbers, it thinks that index 1 is the right class or class number 1 for this particular sample.
13:21:30
And then for this next one, what's the maximum number here? I think it would be the 0th index and the same for the next one. What's the maximum number here? Well, it would be the 0th index as well. But of course, these numbers are going to change once we've trained our model. So how do we get the maximum index value of all of these? So this is where we can go, convert our model's prediction probabilities to prediction labels.
13:22:02
So let's do that. We can go ypreds equals torch dot argmax on ypredprobs. And if we go across the first dimension as well. So now let's have a look at ypreds. Do we have prediction labels in the same format as our ylob test? Beautiful. Yes, we do. Although many of them are wrong, as you can see, ideally they would line up with each other.
13:22:36
But because our model is predicting or making predictions with random numbers, so they haven't been our model hasn't been trained. All of these are basically random outputs. So hopefully once we train our model, it's going to line up the values of the predictions are going to line up with the values of the test labels. But that is how we go from our model's raw outputs to prediction probabilities to prediction labels for a multi-class classification problem.
13:23:03
So let's just add the steps here, logits, raw output of the model, predprobs, to get the prediction probabilities, use torch dot softmax or the softmax activation function, pred labels, take the argmax of the prediction probabilities. So we're going to see this in action later on when we evaluate our model, but I feel like now that we know how to go from logits to prediction probabilities to pred labels, we can write a training loop.
13:23:37
So let's set that up. 8.5, create a training loop, and testing loop for a multi-class pytorch model. This is so exciting. I'll see you in the next video. Let's build our first training and testing loop for a multi-class pytorch model, and I'll give you a little hint. It's quite similar to the training and testing loops we've built before, so you might want to give it a shot. I think you can. Otherwise, we'll do it together in the next video.
13:24:09
Welcome back. In the last video, we covered how to go from raw logits, which is the output of the model, the raw output of the model for a multi-class pytorch model. Then we turned our logits into prediction probabilities using torch.softmax, and then we turn those prediction probabilities into prediction labels by taking the argmax, which returns the index of where the maximum value occurs in the prediction probability.
13:24:38
So for this particular sample, with these four values, because it outputs four values, because we're working with four classes, if we were working with 10 classes, it would have 10 values, the principle of these steps would still be the same. So for this particular sample, this is the value that's the maximum, so we would take that index, which is 1. For this one, the index 0 has the maximum value.
13:25:03
For this sample, same again, and then same again, I mean, these prediction labels are just random, right? So they're quite terrible. But now we're going to change that, because we're going to build a training and testing loop for our multi-class model. Let's do that. So fit the multi-class model to the data. Let's go set up some manual seeds.
13:25:29
Torch dot manual seed, again, don't worry too much if our numbers on the page are not exactly the same. That's inherent to the randomness of machine learning. We're setting up the manual seeds to try and get them as close as possible, but these do not guarantee complete determinism, which means the same output. But we're going to try. The direction is more important.
13:25:51
Set number of epochs. We're going to go epochs. How about we just do 100? I reckon we'll start with that. We can bump it up to 1000 if we really wanted to. Let's put the data to the target device. What's our target device? Well, it doesn't really matter because we've set device agnostic code. So whether we're working with a CPU or a GPU, our code will use whatever device is available. I'm typing blog again.
13:26:18
So we've got x blob train, y blob train. This is going to go where? It's going to go to the device. And y blob train to device. And we're going to go x blob test. And then y blob test equals x blob test to device. Otherwise, we'll get device issues later on, and we'll send this to device as well. Beautiful. Now, what do we do now? Well, we loop through data. Loop through data. So for an epoch in range epochs for an epoch in a range.
13:27:01
Epox. I don't want that auto correct. Come on, Google Colab. Work with me here. We're training our first multi-class classification model. This is serious business. No, I'm joking. It's actually quite fun. So model four dot train. And let's do the forward pass. I'm not going to put much commentary here because we've been through this before. But what are the logits? The logits are raw outputs of our model. So we'll just go x blob train.
13:27:29
And x test. I didn't want that. X blob train. Why did that do that? I need to turn off auto correct in Google Colab. I've been saying it for a long time. Y pred equals torch dot softmax. So what are we doing here? We're going from logits to prediction probabilities here. So torch softmax. Y logits. Across the first dimension. And then we can take the argmax of this and dim equals one.
13:28:03
In fact, I'm going to show you a little bit of, oh, I've written blog here. Maybe auto correct would have been helpful for that. A little trick. You don't actually have to do the torch softmax. The logits. If you just took the argmax of the logits is a little test for you. Just take the argmax of the logits. And see, do you get the same similar outputs as what you get here?
13:28:23
So I've seen that done before, but for completeness, we're going to use the softmax activation function because you'll often see this in practice. And now what do we do? We calculate the loss. So the loss FM. We're going to use categorical cross entropy here or just cross entropy loss. So if we check our loss function, what do we have? We have cross entropy loss. We're going to compare our models, logits to y blob train.
13:28:52
And then what are we going to do? We're going to calculate the accuracy because we're working with the classification problem. It'd be nice if we had accuracy as well as loss. Accuracy is one of the main classification evaluation metrics. y pred equals y pred. y pred. And now what do we do? Well, we have to zero grab the optimizer. Optimizer zero grad. Then we go loss backward. And then we step the optimizer. Optimizer step, step, step.
13:29:25
So none of these steps we haven't covered before. We do the forward pass. We calculate the loss and any evaluation metric we choose to do so. We zero the optimizer. We perform back propagation on the loss. And we step the optimizer. The optimizer will hopefully behind the scenes update the parameters of our model to better represent the patterns in our training data. And so we're going to go testing code here. What do we do for testing code? Well, or inference code.
13:29:53
We set our model to a vowel mode. That's going to turn off a few things behind the scenes that our model doesn't need such as dropout layers, which we haven't covered. But you're more than welcome to check them out if you go torch and end. Dropout layers. Do we have dropout? Dropout layers. Beautiful. And another one that it turns off is match norm. Beautiful. And also you could search this. What does model dot a vowel do?
13:30:24
And you might come across stack overflow question. One of my favorite resources. So there's a little bit of extra curriculum. But I prefer to see things in action. So with torch inference mode, again, this turns off things like gradient tracking and a few more things. So we get as fast as code as possible because we don't need to track gradients when we're making predictions. We just need to use the parameters that our model has learned.
13:30:49
We want X blob test to go to our model here for the test logits. And then for the test preds, we're going to do the same step as what we've done here. We're going to go torch dot softmax on the test logits across the first dimension. And we're going to call the argmax on that to get the index value of where the maximum prediction probability value occurs. We're going to calculate the test loss or loss function. We're going to pass in what the test logits here.
13:31:19
Then we're going to pass in why blob test compare the test logits behind the scenes. Our loss function is going to do some things that convert the test logits into the same format as our test labels and then return us a value for that. Then we'll also calculate the test accuracy here by passing in the why true as why blob test. And we have the y pred equals y pred. Wonderful. And then what's our final step?
13:31:46
Well, we want to print out what's happening because I love seeing metrics as our model trains. It's one of my favorite things to watch. If we go if epoch, let's do it every 10 epochs because we've got 100 so far. It equals zero. Let's print out a nice f string with epoch. And then we're going to go loss. What do we put in here? We'll get our loss value, but we'll take it to four decimal places and we'll get the training accuracy, which will be acc.
13:32:20
And we'll take this to two decimal places and we'll get a nice percentage sign there. And we'll go test loss equals test loss and we'll go there. And finally, we'll go test act at the end here, test act. Now, I'm sure by now we've written a fair few of these. You're either getting sick of them or you're like, wow, I can actually do the steps through here. And so don't worry, we're going to be functionalizing all of this later on,
13:32:49
but I thought I'm going to include them as much as possible so that we can practice as much as possible together. So you ready? We're about to train our first multi-class classification model. In three, two, one, let's go. No typos. Of course. What do we get wrong here? Oh, this is a fun error. Runtime error. NLL loss for reduced CUDA kernel to the index not implemented for float. Okay, that's a pretty full on bunch of words there. I don't really know how to describe that.
13:33:21
But here's a little hint. We've got float there. So we know that float is what? Float is a form of data. It's a data type. So potentially because that's our hint. We said not implemented for float. So maybe we've got something wrong up here. Our data is of the wrong type. Can you see anywhere where our data might be the wrong type? Well, let's print it out. Where's our issue here? Why logits? Why blob train? Okay. Why blob train? And why logits? What does why blob train look like?
13:33:57
Why blob train? Okay. And what's the D type here? Float. Okay. So it's not implemented for float. Hmm. Maybe we have to turn them into a different data type. What if we went type torch long tensor? What happens here? Expected all tensors to be on the same device but found at least two devices. Oh, my goodness. What do we got wrong here? Type torch long tensor. Friends. Guess what? I found it. And so it was to do with this pesky little data type issue here.
13:34:44
So if we run this again and now this one took me a while to find and I want you to know that, that behind the scenes, even though, again, this is a machine learning cooking show, it still takes a while to troubleshoot code and you're going to come across this. But I thought rather than spend 10 minutes doing it in a video, I'll show you what I did. So we went through this and we found that, hmm, there's something going on here. I don't quite know what this is.
13:35:06
And that seems quite like a long string of words, not implemented for float. And then we looked back at the line where it went wrong. And so that we know that maybe the float is hinting at that one of these two tensors is of the wrong data type. Now, why would we think that it's the wrong data type? Well, because anytime you see float or int or something like that, it generally hints at one of your data types being wrong.
13:35:32
And so the error is actually right back up here where we created our tensor data. So we turned our labels here into float, which generally is okay in PyTorch. However, this one should be of type torch dot long tensor, which we haven't seen before. But if we go into torch long tensor, let's have a look torch dot tensor. Do we have long tensor? Here we go. 64 bit integer signed. So why do we need torch dot long tensor? And again, this took me a while to find.
13:36:10
And so I want to express this that in your own code, you probably will butt your head up against some issues and errors that do take you a while to find. And data types is one of the main ones. But if we look in the documentation for cross entropy loss, the way I kind of found this out was this little hint here. The performance of the criteria is generally better when the target contains class indices, as this allows for optimized computation.
13:36:36
But I read this and it says target contains class indices. I'm like, hmm, alza indices already, but maybe they should be integers and not floats. But then if you actually just look at the sample code, you would find that they use d type equals torch dot long. Now, that's the thing with a lot of code around the internet is that sometimes the answer you're looking for is a little bit buried. But if in doubt, run the code and butt your head up against a wall for a bit and keep going.
13:37:10
So let's just rerun all of this and see do we have an error here? Let's train our first multi-class classification model together. No arrows, fingers crossed. But what did we get wrong here? OK, so we've got different size. We're slowly working through all of the errors in deep learning here. Value error, input batch size 200 to match target size 200. So this is telling me maybe our test data, which is of size 200, is getting mixed up with our training data, which is of size 800.
13:37:40
So we've got test loss, the test logits, model four. What's the size? Let's print out print test logits dot shape and wine blob test. So troubleshooting on the fly here, everyone. What do we got? Torch size 800. Where are our test labels coming from? Wine blob test equals, oh, there we go. Ah, did you catch that before? Maybe you did, maybe you didn't. But I think we should be right here. Now if we just comment out this line, so we've had a data type issue and we've had a shape issue.
13:38:24
Two of the main and machine learning, oh, and again, we've had some issues. Wine blob test. What's going on here? I thought we just changed the shape. Oh, no, we have to go up and reassign it again because now this is definitely why blob, yes. Let's rerun all of this, reassign our data. We are running into every single error here, but I'm glad we're doing this because otherwise you might not see how to troubleshoot these type of things. So the size of a tensor much match the size.
13:38:59
Oh, we're getting the issue here. Test spreads. Oh, my goodness. We have written so much code here. Test spreads. So instead of wire spread, this should be test spreads. Fingers crossed. Are we training our first model yet or what? There we go. Okay, I'm going to printing out some stuff. I don't really want to print out that stuff. I want to see the loss go down, so I'm going to. So friends, I hope you know that we've just been through some of the most fundamental troubleshooting steps.
13:39:29
And you might say, oh, Daniel, there's a cop out because you're just coding wrong. And in fact, I code wrong all the time. But we've now worked out how to troubleshoot them shape errors and data type errors. But look at this. After all of that, thank goodness. Our loss and accuracy go in the directions that we want them to go. So our loss goes down and our accuracy goes up. Beautiful. So it looks like that our model is working on a multi-class classification data set. So how do we check that?
13:40:03
Well, we're going to evaluate it in the next step by visualize, visualize, visualize. So you might want to give that a shot. See if you can use our plot decision boundary function. We'll use our model to separate the data here. So it's going to be much the same as what we did for binary classification. But this time we're using a different model and a different data set. I'll see you there. Welcome back. In the last video, we went through some of the steps that we've been through before
13:40:33
in terms of training and testing a model. But we also butted our heads up against two of the most common issues in machine learning and deep learning in general. And that is data type issues and shape issues. But luckily we were able to resolve them. And trust me, you're going to run across many of them in your own deep learning and machine learning endeavors. So I'm glad that we got to have a look at them and sort of I could show you what I do to troubleshoot them.
13:40:59
But in reality, it's a lot of experimentation. Run the code, see what errors come out, Google the errors, read the documentation, try again. But with that being said, it looks like that our model, our multi-class classification model has learned something. The loss is going down, the accuracy is going up. But we can further evaluate this by making and evaluating predictions with a PyTorch multi-class model. So how do we make predictions? We've seen this step before, but let's reiterate.
13:41:36
Make predictions, we're going to set our model to what mode, a vowel mode. And then we're going to turn on what context manager, inference mode. Because we want to make inference, we want to make predictions. Now what do we make predictions on? Or what are the predictions? They're going to be logits because why? They are the raw outputs of our model. So we'll take model four, which we just trained and we'll pass it the test data. Well, it needs to be blob test, by the way.
13:42:02
I keep getting that variable mixed up. We just had enough problems with the data, Daniel. We don't need any more. You're completely right. I agree with you. But we're probably going to come across some more problems in the future. Don't you worry about that. So let's view the first 10 predictions. Why logits? What do they look like? All right, just numbers on the page. They're raw logits. Now how do we go from go from logits to prediction probabilities? How do we do that?
13:42:32
With a multi-class model, we go y-pred-probs equals torch.softmax on the y-logits. And we want to do it across the first dimension. And what do we have when we go pred-probs? Let's go up to the first 10. Are we apples to apples yet? What does our y-blog test look like? We're not apples to apples yet, but we're close. So these are prediction probabilities. You'll notice that we get some fairly different values here. And remember, the one closest to one here, the value closest to one,
13:43:08
which looks like it's this, the index of the maximum value is going to be our model's predicted class. So this index is index one. And does it correlate here? Yes. One beautiful. Then we have index three, which is the maximum value here. Three beautiful. And then we have, what do we have here? Index two, yes. Okay wonderful. But let's not step through that. We're programmers. We can do this with code. So now let's go from pred-probs to pred-labels. So y-pred-equals, how do we do that?
13:43:44
Well, we can do torch.argmax on the y-pred-probs. And then we can pass dim equals one. We could also do it this way. So y-pred-probs call dot-argmax. There's no real difference between these two. But we're just going to do it this way, called torch.argmax. y-pred-es, let's view the first 10. Are we now comparing apples to apples when we go y-blob test? Yes, we are. Have a go at that. Look one three two one zero three one three two one zero three. Beautiful.
13:44:21
Now, we could line these up and look at and compare them all day. I mean, that would be fun. But I know what something that would be even more fun. Let's get visual. So plot dot figure. And we're going to go fig size equals 12.6, just because the beauty of this being a cooking show is I kind of know what ingredients work from ahead of time. Despite what you saw in the last video with all of that trouble shooting. But I'm actually glad that we did that because seriously.
13:44:50
Shape issues and data type issues. You're going to come across a lot of them. The two are the main issues I troubleshoot, aside from device issues. So let's go x-blob train and y-blob train. And we're going to do another plot here. We're going to get subplot one, two, two. And we're going to do this for the test data. Test and then plot decision boundary. Plot decision boundary with model four on x-blob test and y-blob test as well. Let's see this. Did we train a multi-class? Oh my goodness.
13:45:33
Yes, we did. Our code worked faster than I can speak. Look at that beautiful looking plot. We've separated our data almost as best as what we could. Like there's some here that are quite inconspicuous. And now what's the thing about these lines? With this model have worked, I posed the question a fair few videos ago, whenever we created our multi-class model that could we separate this data without nonlinear functions. So how about we just test that?
13:46:01
Since we've got the code ready, let's go back up. We've got nonlinear functions here. We've got relu here. So I'm just going to recreate our model there. So I just took relu out. That's all I did. Commented it out, this code will still all work. Or fingers crossed it will. Don't count your chickens before they hatch. Daniel, come on. We're just going to rerun all of these cells. All the code's going to stay the same. All we did was we took the nonlinearity out of our model.
13:46:26
Is it still going to work? Oh my goodness. It still works. Now why is that? Well, you'll notice that the lines are a lot more straighter here. Did we get different metrics? I'll leave that for you to compare. Maybe these will be a little bit different. I don't think they're too far different. But that is because our data is linearly separable. So we can draw straight lines only to separate our data. However, a lot of the data that you deal with in practice will require linear and nonlinear.
13:46:57
Hence why we spent a lot of time on that. Like the circle data that we covered before. And let's look up an image of a pizza. If you're building a food vision model to take photos of food and separate different classes of food, could you do this with just straight lines? You might be able to, but I personally don't think that I could build a model to do such a thing. And in fact, PyTorch makes it so easy to add nonlinearities to our model, we might as well have them in
13:47:24
so that our model can use it if it needs it and if it doesn't need it, well, hey, it's going to build a pretty good model as we saw before if we included the nonlinearities in our model. So we could uncomment these and our model is still going to perform quite well. That is the beauty of neural networks, is that they decide the numbers that should represent outdated the best. And so, with that being said, we've evaluated our model, we've trained our multi-class classification model,
13:47:51
we've put everything together, we've gone from binary classification to multi-class classification. I think there's just one more thing that we should cover and that is, let's go here, section number nine, a few more classification metrics. So, as I said before, evaluating a model, let's just put it here, to evaluate our model, our classification models, that is, evaluating a model is just as important as training a model. So, I'll see you in the next video.
13:48:24
Let's cover a few more classification metrics. Welcome back. In the last video, we evaluated our multi-class classification model visually. And we saw that it did pretty darn well, because our data turned out to be linearly separable. So, our model, even without non-linear functions, could separate the data here. However, as I said before, most of the data that you deal with will require some form of linear and non-linear function. So, just keep that in mind, and the beauty of PyTorch is
13:48:53
that it allows us to create models with linear and non-linear functions quite flexibly. So, let's write down here. If we wanted to further evaluate our classification models, we've seen accuracy. So, accuracy is one of the main methods of evaluating classification models. So, this is like saying, out of 100 samples, how many does our model get right? And so, we've seen our model right now is that testing accuracy of nearly 100%. So, it's nearly perfect.
13:49:25
But, of course, there were a few tough samples, which I mean a little bit hard. Some of them are even within the other samples, so you can forgive it a little bit here for not being exactly perfect. What are some other metrics here? Well, we've also got precision, and we've also got recall. Both of these will be pretty important when you have classes with different amounts of values in them. So, precision and recall. So, accuracy is pretty good to use when you have balanced classes.
13:49:57
So, this is just text on a page for now. F1 score, which combines precision and recall. There's also a confusion matrix, and there's also a classification report. So, I'm going to show you a few code examples of where you can access these, and I'm going to leave it to you as extra curriculum to try each one of these out. So, let's go into the keynote. And by the way, you should pay yourself on the back here because we've just gone through all of the PyTorch workflow for a classification problem.
13:50:30
Not only just binary classification, we've done multi-class classification as well. So, let's not stop there, though. Remember, building a model, evaluating a model is just as important as building a model. So, we've been through non-linearity. We've seen how we could replicate non-linear functions. We've talked about the machine learning explorer's motto, visualize visualize visualize. Machine learning practitioners motto is experiment, experiment, experiment.
13:50:58
I think I called that the machine learning or data scientist motto. Same thing, you know? And steps in modeling with PyTorch. We've seen this in practice, so we don't need to look at these slides. I mean, they'll be available on the GitHub if you want them, but here we are. Some common classification evaluation methods. So, we have accuracy. There's the formal formula if you want, but there's also code, which is what we've been focusing on.
13:51:20
So, we wrote our own accuracy function, which replicates this. By the way, Tp stands for not toilet paper, it stands for true positive, Tn is true negative, false positive, Fp, false negative, Fn. And so, the code, we could do torch metrics. Oh, what's that? But when should you use it? The default metric for classification problems. Note, it is not the best for imbalanced classes. So, if you had, for example, 1,000 samples of one class, so, number one, label number one,
13:51:50
but you had only 10 samples of class zero. So, accuracy might not be the best to use for then. For imbalanced data sets, you might want to look into precision and recall. So, there's a great article called, I think it's beyond accuracy, precision and recall, which gives a fantastic overview of, there we go. This is what I'd recommend. There we go, by Will Coestron. So, I'd highly recommend this article as some extra curriculum here. See this article for when to use precision recall.
13:52:29
We'll go there. Now, if we look back, there is the formal formula for precision, true positive over true positive plus false positive. So, higher precision leads to less false positives. So, if false positives are not ideal, you probably want to increase precision. If false negatives are not ideal, you want to increase your recall metric. However, you should be aware that there is such thing as a precision recall trade-off. And you're going to find this in your experimentation.
13:52:58
Precision recall trade-off. So, that means that, generally, if you increase precision, you lower recall. And, inversely, if you increase precision, you lower recall. So, check out that, just to be aware of that. But, again, you're going to learn this through practice of evaluating your models. If you'd like some code to do precision and recall, you've got torchmetrics.precision, or torchmetrics.recall, as well as scikit-learn.
13:53:26
So scikit-learn has implementations for many different classification metrics. Torchmetrics is a PyTorch-like library. And then we have F1 score, which combines precision and recall. So, it's a good combination if you want something in between these two. And then, finally, there's a confusion matrix. I haven't listed here a classification report, but I've listed it up here. And we can see a classification report in scikit-learn. Classification report.
13:53:55
Classification report kind of just puts together all of the metrics that we've talked about. And we can go there. But I've been talking a lot about torchmetrics. So let's look up torchmetrics' accuracy. Torchmetrics. So this is a library. I don't think it comes with Google Colab at the moment, but you can import torchmetrics, and you can initialize a metric. So we've built our own accuracy function, but the beauty of using torchmetrics is that it uses PyTorch-like code.
13:54:27
So we've got metric, preds, and target. And then we can find out what the value of the accuracy is. And if you wanted to implement your own metrics, you could subclass the metric class here. But let's just practice this. So let's check to see if I'm going to grab this and copy this in here. If you want access to a lot of PyTorch metrics, see torchmetrics. So can we import torchmetrics? Maybe it's already in Google Colab. No, not here. But that's all right. We'll go pip install torchmetrics.
13:55:13
So Google Colab has access to torchmetrics. And that's going to download from torchmetrics. It shouldn't take too long. It's quite a small package. Beautiful. And now we're going to go from torchmetrics import accuracy. Wonderful. And let's see how we can use this. So setup metric. So we're going to go torchmetric underscore accuracy. We could call it whatever we want, really. But we need accuracy here. We're just going to set up the class.
13:55:44
And then we're going to calculate the accuracy of our multi-class model by calling torchmetric accuracy. And we're going to pass it Y threads and Y blob test. Let's see what happens here. Oh, what did we get wrong? Runtime error. Expected all tensors to be on the same device, but found at least two devices. Oh, of course. Now, remember how I said torchmetrics implements PyTorch like code? Well, let's check what device this is on. Oh, it's on the CPU.
13:56:22
So something to be aware of that if you use torchmetrics, you have to make sure your metrics are on the same device by using device agnostic code as your data. So if we run this, what do we get? Beautiful. We get an accuracy of 99.5%, which is in line with the accuracy function that we coded ourselves. So if you'd like a lot of pre-built metrics functions, be sure to see either scikit-learn for any of these or torchmetrics for any PyTorch like metrics.
13:56:53
But just be aware, if you use the PyTorch version, they have to be on the same device. And if you'd like to install it, what do we have? Where's the metrics? Module metrics? Do we have classification? There we go. So look how many different types of classification metrics there are from torchmetrics. So I'll leave that for you to explore. The resources for this will be here. This is an extracurricular article for when to use precision recall.
13:57:20
And another extracurricular would be to go through the torchmetrics module for 10 minutes and have a look at the different metrics for classification. So with that being said, I think we've covered a fair bit. But I think it's also time for you to practice what you've learned. So let's cover some exercises in the next video. I'll see you there. Welcome back. In the last video, we looked at a few more classification metrics, a little bit of a high
13:57:52
level overview for some more ways to evaluate your classification models. And I linked some extracurricular here that you might want to look into as well. But we have covered a whole bunch of code together. But now it's time for you to practice some of this stuff on your own. And so I have some exercises prepared. Now, where do you go for the exercises? Well, remember on the learnpytorch.io book, for each one of these chapters, there's a section. Now, just have a look at how much we've covered.
13:58:19
If I scroll, just keep scrolling. Look at that. We've covered all of that in this module. That's a fair bit of stuff. But down the bottom of each one is an exercises section. So all exercises are focusing on practicing the code in the sections above, all of these sections here. I've got number one, two, three, four, five, six, seven. Yeah, seven exercises, nice, writing plenty of code. And then, of course, extracurricular.
13:58:44
So these are some challenges that I've mentioned throughout the entire section zero two. But I'm going to link this in here. Exercises. But, of course, you can just find it on the learnpytorch.io book. So if we come in here and we just create another heading. Exercises. And extracurricular. And then we just write in here. See exercises and extracurricular. Here. And so if you'd like a template of the exercise code, you can go to the PyTorch deep learning repo.
13:59:28
And then within the extras folder, we have exercises and solutions. You might be able to guess what's in each of these exercises. We have O2 PyTorch classification exercises. This is going to be some skeleton code. And then, of course, we have the solutions as well. Now, this is just one form of solutions. But I'm not going to look at those because I would recommend you looking at the exercises first before you go into the solutions. So we have things like import torch.
13:59:57
Set up device agnostic code. Create a data set. Turn data into a data frame. And then et cetera, et cetera. For the things that we've done throughout this section. So give that a go. Try it on your own. And if you get stuck, you can refer to the notebook that we've coded together. All of this code here. You can refer to the documentation, of course. And then you can refer to as a last resort, the solutions notebooks. So give that a shot. And congratulations on finishing.
14:00:31
Section 02 PyTorch classification. Now, if you're still there, you're still with me. Let's move on to the next section. We're going to cover a few more things of deep learning with PyTorch. I'll see you there. Hello, and welcome back. We've got another section. We've got computer vision and convolutional neural networks with. PyTorch. Now, computer vision is one of my favorite, favorite deep learning topics. But before we get into the materials, let's answer a very important question.
14:01:06
And that is, where can you get help? So, first and foremost, is to follow along with the code as best you can. We're going to be writing a whole bunch of PyTorch computer vision code. And remember our motto. If and out, run the code. See what the inputs and outputs are of your code. And that's try it yourself. If you need the doc string to read about what the function you're using does, you can press shift command and space in Google CoLab. Or it might be control if you're on Windows.
14:01:33
Otherwise, if you're still stuck, you can search for the code that you're running. You might come across stack overflow or the PyTorch documentation. We've spent a bunch of time in the PyTorch documentation already. And we're going to be referencing a whole bunch in the next module in section three. We're up to now. If you go through all of these four steps, the next step is to try it again. If and out, run the code.
14:01:55
And then, of course, if you're still stuck, you can ask a question on the PyTorch deep learning repo. Discussions tab. Now, if we open this up, we can go new discussion. And you can write here section 03 for the computer vision. My problem is, and then in here, you can write some code. Be sure to format it as best you can. That way it'll help us answer it. And then go, what's happening here? Now, why do I format the code in these back ticks here?
14:02:27
It's so that it looks like code and that it's easier to read when it's formatted on the GitHub discussion. Then you can select a category. If you have a general chat, an idea, a poll, a Q&A, or a show and tell of something you've made, or what you've learned from the course. For question and answering, you want to put it as Q&A. Then you can click start discussion. And it'll appear here. And that way, they'll be searchable and we'll be able to help you out. So I'm going to get out of this.
14:02:52
And oh, speaking of resources, we've got the PyTorch deep learning repo. The links will be where you need the links. All of the code that we're going to write in this section is contained within the section 3 notebook. PyTorch computer vision. Now, this is a beautiful notebook annotated with heaps of text and images. You can go through this on your own time and use it as a reference to help out.
14:03:13
If you get stuck on any of the code we write through the videos, check it out in this notebook because it's probably here somewhere. And then finally, let's get out of these. If we come to the book version of the course, this is learnpytorch.io. We've got home. This will probably be updated by the time you look at that. But we have section 03, which is PyTorch computer vision. It's got all of the information about what we're going to cover in a book format.
14:03:39
And you can, of course, skip ahead to different subtitles. See what we're going to write here. All of the links you need and extra resources will be at learnpytorch.io. And for this section, it's PyTorch computer vision. With that being said, speaking of computer vision, you might have the question, what is a computer vision problem? Well, if you can see it, it could probably be phrased at some sort of computer vision problem. That's how broad computer vision is.
14:04:08
So let's have a few concrete examples. We might have a binary classification problem, such as if we wanted to have two different images. Is this photo of steak or pizza? We might build a model that understands what steak looks like in an image. This is a beautiful dish that I cooked, by the way. This is me eating pizza at a cafe with my dad. And so we could have binary classification, one thing or another. And so our machine learning model may take in the pixels of an image
14:04:35
and understand the different patterns that go into what a steak looks like and the same thing with a pizza. Now, the important thing to note is that we won't actually be telling our model what to learn. It will learn those patterns itself from different examples of images. Then we could step things up and have a multi-class classification problem. You're noticing a trend here. We've covered classification before, but classification can be quite broad.
14:05:00
It can be across different domains, such as vision or text or audio. But if we were working with multi-class classification for an image problem, we might have, is this a photo of sushi, steak or pizza? And then we have three classes instead of two. But again, this could be 100 classes, such as what Nutrify uses, which is an app that I'm working on. We go to Nutrify.app. This is bare bones at the moment. But right now, Nutrify can classify up to 100 different foods.
14:05:29
So if you were to upload an image of food, let's give it a try. Nutrify, we'll go into images, and we'll go into sample food images. And how about some chicken wings? What does it classify this as? Chicken wings. Beautiful. And then if we upload an image of not food, maybe. Let's go to Nutrify. This is on my computer, by the way. You might not have a sample folder set up like this. And then if we upload a photo of a Cybertruck, what does it say? No food found. Please try another image.
14:06:00
So behind the scenes, Nutrify is using the pixels of an image and then running them through a machine learning model and classifying it first, whether it's food or not food. And then if it is food, classifying it as what food it is. So right now it works for 100 different foods. So if we have a look at all these, it can classify apples, artichokes, avocados, barbecue sauce. Each of these work at different levels of performance, but that's just something to keep in mind of what you can do.
14:06:27
So the whole premise of Nutrify is to upload a photo of food and then learn about the nutrition about it. So let's go back to our keynote. What's another example? Well, we could use computer vision for object detection, where you might answer the question is, where's the thing we're looking for? So for example, this car here, I caught them on security camera, actually did a hit and run on my new car, wasn't that much of an expensive car, but I parked it on the street
14:06:54
and this person, the trailer came off the back of their car and hit my car and then they just picked the trailer up and drove away. But I went to my neighbor's house and had to look at their security footage and they found this car. So potentially, you could design a machine learning model to find this certain type of car. It was an orange jute, by the way, but the images were in black and white to detect to see if it ever recognizes a similar car that comes across the street and you could go,
14:07:21
hey, did you crash into my car the other day? I didn't actually find who it was. So sadly, it was a hit and run. But that's object detection, finding something in an image. And then you might want to find out whether the different sections in this image. So this is a great example at what Apple uses on their devices, iPhones and iPads and whatnot, to segregate or segment the different sections of an image, so person one, person two, skin tones, hair, sky, original.
14:07:49
And then it enhances each of these sections in different ways. So that's a practice known as computational photography. But the whole premise is how do you segment different portions of an image? And then there's a great blog post here that talks about how it works and what it does and what kind of model that's used. I'll leave that as extra curriculum if you'd like to look into it. So if you have these images, how do you enhance the sky? How do you make the skin tones look how they should?
14:08:16
How do you remove the background if you really wanted to? So all of this happens on device. So that's where I got that image from, by the way. Semantic Mars. And this is another great blog, Apple Machine Learning Research. So to keep this in mind, we're about to see another example for computer vision, which is Tesla Computer Vision. A lot of companies have websites such as Apple Machine Learning Research where they share a whole bunch of what they're up to in the world of machine learning.
14:08:44
So in Tesla's case, they have eight cameras on each of their self-driving cars that fuels their full self-driving beta software. And so they use computer vision to understand what's going on in an image and then plan what's going on. So this is three-dimensional vector space. And what this means is they're basically taking these different viewpoints from the eight different cameras, feeding them through some form of neural network,
14:09:08
and turning the representation of the environment around the car into a vector. So a long string of numbers. And why will it do that? Well, because computers understand numbers far more than they understand images. So we might be able to recognize what's happening here. But for a computer to understand it, we have to turn it into vector space. And so if you want to have a look at how Tesla uses computer vision, so this is from Tesla's AI Day video.
14:09:33
I'm not going to play it all because it's three hours long, but I watched it and I really enjoyed it. So there's some information there. And there's a little tidbit there. If you go to two hours and one minute and 31 seconds on the same video, have a look at what Tesla do. Well, would you look at that? Where have we seen that before? That's some device-agnostic code, but with Tesla's custom dojo chip. So Tesla uses PyTorch. So the exact same code that we're writing,
14:10:00
Tesla uses similar PyTorch code to, of course, they write PyTorch code to suit their problem. But nonetheless, they use PyTorch code to train their machine learning models that power their self-driving software. So how cool is that? And if you want to have a look at another example, there's plenty of different Tesla self-driving videos. So, oh, we can just play it right here. I was going to click the link. So look, this is what happens. If we have a look in the environment,
14:10:26
Tesla, the cameras, understand what's going on here. And then it computes it into this little graphic here on your heads-up display in the car. And it kind of understands, well, I'm getting pretty close to this car. I'm getting pretty close to that car. And then it uses this information about what's happening, this perception, to plan where it should drive next. And I believe here it ends up going into it. It has to stop. Yeah, there we go. Because we've got a stop sign. Look at that.
14:10:55
It's perceiving the stop sign. It's got two people here. It just saw a car drive pass across this street. So that is pretty darn cool. That's just one example of computer vision, one of many. And how would you find out what computer vision can be used for? Here's what I do. What can computer vision be used for? Plenty more resources. So, oh, there we go. 27 most popular computer vision applications in 2022. So we've covered a fair bit there.
14:11:22
But what are we going to cover specifically with PyTorch code? Well, broadly, like that. We're going to get a vision data set to work with using torch vision. So PyTorch has a lot of different domain libraries. Torch vision helps us deal with computer vision problems. And there's existing data sets that we can leverage to play around with computer vision. We're going to have a look at the architecture of a convolutional neural network, also known as a CNN with PyTorch.
14:11:47
We're going to look at an end-to-end multi-class image classification problem. So multi-class is what? More than one thing or another? Could be three classes, could be a hundred. We're going to look at steps at modeling with CNNs in PyTorch. So we're going to create a convolutional network with PyTorch. We're going to pick a last function and optimize it to suit our problem. We're going to train a model, training a model a model. A little bit of a typo there.
14:12:10
And then we're going to evaluate a model, right? So we might have typos with our text, but we'll have less typos in the code. And how are we going to do this? Well, we could do it cook, so we could do it chemis. Well, we're going to do it a little bit of both. Part art, part science. But since this is a machine learning cooking show, we're going to be cooking up lots of code. So in the next video, we're going to cover the inputs and outputs of a computer vision problem. I'll see you there.
14:12:36
So in the last video, we covered what we're going to cover, broadly. And we saw some examples of what computer vision problems are. Essentially, anything that you're able to see, you can potentially turn into a computer vision problem. And we're going to be cooking up lots of machine learning, or specifically pie torch, computer vision code. You see I fixed that typo. Now let's talk about what the inputs and outputs are of a typical computer vision problem.
14:13:00
So let's start with a multi-classification example. And so we wanted to take photos of different images of food and recognize what they were. So we're replicating the functionality of Nutrify. So take a photo of food and learn about it. So we might start with a bunch of food images that have a height and width of some sort. So we have width equals 224, height equals 224, and then they have three color channels. Why three? Well, that's because we have a value for red, green and blue.
14:13:32
So if we look at this up, if we go red, green, blue image format. So 24-bit RGB images. So a lot of images or digital images have some value for a red pixel, a green pixel and a blue pixel. And if you were to convert images into numbers, they get represented by some value of red, some value of green and some value of blue. That is exactly the same as how we'd represent these images.
14:14:02
So for example, this pixel here might be a little bit more red, a little less blue, and a little less green because it's close to orange. And then we convert that into numbers. So what we're trying to do here is essentially what we're trying to do with all of the data that we have with machine learning is represented as numbers. So the typical image format to represent an image because we're using computer vision. So we're trying to figure out what's in an image.
14:14:25
The typical way to represent that is in a tensor that has a value for the height, width and color channels. And so we might numerically encode these. In other words, represent our images as a tensor. And this would be the inputs to our machine learning algorithm. And in many cases, depending on what problem you're working on, an existing algorithm already exists for many of the most popular computer vision problems. And if it doesn't, you can build one.
14:14:51
And then you might fashion this machine learning algorithm to output the exact shapes that you want. In our case, we want three outputs. We want one output for each class that we have. We want a prediction probability for sushi. We want a prediction probability for steak. And we want a prediction probability for pizza. Now in our case, in this iteration, looks like our model got one of them wrong because the highest value was assigned to the wrong class here.
14:15:17
So for the second image, it assigned a prediction probability of 0.81 for sushi. Now, keep in mind that you could change these classes to whatever your particular problem is. I'm just simplifying this and making it three. You could have a hundred. You could have a thousand. You could have five. It's just, it depends on what you're working with. And so we might use these predicted outputs to enhance our app. So if someone wants to take a photo of their plate of sushi, our app might say,
14:15:46
hey, this is a photo of sushi. Here's some information about those, the sushi rolls or the same for steak, the same for pizza. Now it might not always get it right because after all, that's what machine learning is. It's probabilistic. So how would we improve these results here? Well, we could show our model more and more images of sushi steak and pizza so that it builds up a better internal representation of said images.
14:16:12
So when it looks at images it's never seen before or images outside its training data set, it's able to get better results. But just keep in mind this whole process is similar no matter what computer vision problem you're working with. You need a way to numerically encode your information. You need a machine learning model that's capable of fitting the data in the way that you would like it to be fit in our case classification.
14:16:34
You might have a different type of model if you're working with object detection, a different type of model if you're working with segmentation. And then you need to fashion the outputs in a way that best suit your problem as well. So let's push forward. Oh, by the way, the model that often does this is a convolutional neural network. In other words, a CNN. However, you can use many other different types of machine learning algorithms here.
14:16:58
It's just that convolutional neural networks typically perform the best with image data. Although with recent research, there is the transformer architecture or deep learning model that also performs fairly well or very well with image data. So just keep that in mind going forward. But for now we're going to focus on convolutional neural networks. And so we might have input and output shapes because remember one of the chief machine learning problems
14:17:23
is making sure that your tensor shapes line up with each other, the input and output shapes. So if we encoded this image of stake here, we might have a dimensionality of batch size with height color channels. And now the ordering here could be improved. It's usually height then width. So alphabetical order. And then color channels last. So we might have the shape of none, two, two, four, two, four, three. Now where does this come from? So none could be the batch size.
14:17:50
Now it's none because we can set the batch size to whatever we want, say for example 32. Then we might have a height of two to four and a width of two to four and three color channels. Now height and width are also customizable. You might change this to be 512 by 512. What that would mean is that you have more numbers representing your image. And in sense would take more computation to figure out the patterns because there is simply more information encoded in your image.
14:18:19
But two, two, four, two, four is a common starting point for images. And then 32 is also a very common batch size, as we've seen in previous videos. But again, this could be changed depending on the hardware you're using, depending on the model you're using. You might have a batch size to 64. You might have a batch size of 512. It's all problem specific. And that's this line here. These will vary depending on the problem you're working on.
14:18:44
So in our case, our output shape is three because we have three different classes for now. But again, if you have a hundred, you might have an output shape of a hundred. If you have a thousand, you might have an output shape of a thousand. The same premise of this whole pattern remains though. Numerically encode your data, feed it into a model, and then make sure the output shape fits your specific problem.
14:19:07
And so, for this section, Computer Vision with PyTorch, we're going to be building CNNs to do this part. We're actually going to do all of the parts here, but we're going to focus on building a convolutional neural network to try and find patterns in data, because it's not always guaranteed that it will. Finally, let's look at one more problem. Say you had grayscale images of fashion items, and you have quite small images. They're only 28 by 28. The exact same pattern is going to happen.
14:19:38
You numerically represent it, use it as inputs to a machine learning algorithm, and then hopefully your machine learning algorithm outputs the right type of clothing that it is. In this case, it's a t-shirt. But I've got dot dot dot here because we're going to be working on a problem that uses ten different types of items of clothing. And the images are grayscale, so there's not much detail. So hopefully our machine learning algorithm can recognize what's going on in these images.
14:20:02
There might be a boot, there might be a shirt, there might be pants, there might be a dress, etc etc. But we numerically encode our images into dimensionality of batch size, height with color channels. This is known as NHWC, or number of batches, or number of images in a batch, height with C, or color channels. This is color channels last. Why am I showing you two forms of this? Do you notice color channels in this one is color channels first? So color channels height width?
14:20:33
Well, because you come across a lot of different representations of data full stop, but particularly image data in PyTorch and other libraries, many libraries expect color channels last. However, PyTorch currently at the time of recording this video may change in the future, defaults to representing image data with color channels first. Now this is very important because you will get errors if your dimensionality is in the wrong order.
14:20:59
And so there are ways to go in between these two, and there's a lot of debate of which format is the best. It looks like color channels last is going to win over the long term, just because it's more efficient, but again, that's outside the scope, but just keep this in mind. We're going to write code to interact between these two, but it's the same data just represented in different order. And so we could rearrange these shapes to how we want color channels last or color channels first.
14:21:25
And once again, the shapes will vary depending on the problem that you're working on. So with that being said, we've covered the input and output shapes. How are we going to see them with code? Well, of course we're going to be following the PyTorch workflow that we've done. So we need to get our data ready, turn it into tenses in some way, shape or form. We can do that with taught division transforms. Oh, we haven't seen that one yet, but we will.
14:21:52
We can use torchutilsdata.datasetutils.data.data loader. We can then build a model or pick a pre-trained model to suit our problem. We've got a whole bunch of modules to help us with that, torchNN module, torchvision.models. And then we have an optimizer and a loss function. We can evaluate the model using torch metrics, or we can code our own metric functions. We can of course improve through experimentation, which we will see later on, which we've actually done that, right?
14:22:21
We've done improvement through experimentation. We've tried different models, we've tried different things. And then finally, we can save and reload our trained model if we wanted to use it elsewhere. So with that being said, we've covered the workflow. This is just a high-level overview of what we're going to code. You might be asking the question, what is a convolutional neural network, or a CNN? Let's answer that in the next video. I'll see you there. Welcome back.
14:22:47
In the last video, we saw examples of computer vision input and output shapes. And we kind of hinted at the fact that convolutional neural networks are deep learning models, or CNNs, that are quite good at recognizing patterns in images. So we left off the last video with the question, what is a convolutional neural network? And where could you find out about that? What is a convolutional neural network? Here's one way to find out.
14:23:15
And I'm sure, as you've seen, there's a lot of resources for such things. A comprehensive guide to convolutional neural networks. Which one of these is the best? Well, it doesn't really matter. The best one is the one that you understand the best. So there we go. There's a great video from Code Basics. I've seen that one before, simple explanation of convolutional neural network. I'll leave you to research these things on your own.
14:23:38
And if you wanted to look at images, there's a whole bunch of images. I prefer to learn things by writing code. Because remember, this course is code first. As a machine learning engineer, 99% of my time is spent writing code. So that's what we're going to focus on. But anyway, here's the typical architecture of a CNN. In other words, a convolutional neural network. If you hear me say CNN, I'm not talking about the news website.
14:24:01
In this course, I'm talking about the architecture convolutional neural network. So this is some PyTorch code that we're going to be working towards building. But we have some hyperparameters slash layer types here. We have an input layer. So we have an input layer, which takes some in channels, and an input shape. Because remember, it's very important in machine learning and deep learning to line up your
14:24:23
input and output shapes of whatever model you're using, whatever problem you're working with. Then we have some sort of convolutional layer. Now, what might happen in a convolutional layer? Well, as you might have guessed, as what happens in many neural networks, is that the layers perform some sort of mathematical operation. Now, convolutional layers perform convolving window operation across an image or across a tensor. And discover patterns using, let's have a look, actually.
14:24:53
Let's go, nn.com2d. There we go. This is what happens. So the output of our network equals a bias plus the sum of the weight tensor over the convolutional channel out, okay, times input. Now, if you want to dig deeper into what is actually going on here, you're more than welcome to do that. But we're going to be writing code that leverages the torch nn.com2d. And we're going to fix up all of these hyperparameters here so that it works with our problem.
14:25:29
Now, what you need to know here is that this is a bias term. We've seen this before. And this is a weight matrix. So a bias vector typically and a weight matrix. And they operate over the input. But we'll see these later on with code. So just keep that in mind. This is what's happening. As with every layer in a neural network, some form of operation is happening on our input data. These operations happen layer by layer until eventually, hopefully, they can be turned into some usable output.
14:26:02
So let's jump back in here. Then we have an hidden activation slash nonlinear activation because why do we use nonlinear activations? Well, it's because if our data was nonlinear, non-straight lines, we need the help of straight and non-straight lines to model it, to draw patterns in it. Then we typically have a pooling layer. And I want you to take this architecture. I've said typical here for a reason because these type of architectures are changing all the time.
14:26:32
So this is just one typical example of a CNN. It's about as basic as a CNN as you can get. So over time, you will start to learn to build more complex models. You will not only start to learn to build them, you will just start to learn to use them, as we'll see later on in the transfer learning section of the course. And then we have an output layer. So do you notice the trend here? We have an input layer and then we have multiple hidden layers that perform some sort of mathematical
14:26:55
operation on our data. And then we have an output slash linear layer that converts our output into the ideal shape that we'd like. So we have an output shape here. And then how does this look in process? While we put in some images, they go through all of these layers here because we've used an end up sequential. And then hopefully this forward method returns x in a usable status or usable state that we can convert into class names.
14:27:20
And then we could integrate this into our computer vision app in some way, shape or form. And here's the asterisk here. Note, there are almost an unlimited amount of ways you could stack together a convolutional neural network. This slide only demonstrates one. So just keep that in mind, only demonstrates one. But the best way to practice this sort of stuff is not to stare at a page. It's to if and out, code it out. So let's code, I'll see you in Google CoLab. Welcome back.
14:27:50
Now, we've discussed a bunch of fundamentals about computer vision problems and convolutional neural networks. But rather than talk to more slides, well, let's start to code them out. I'm going to meet you at colab.research.google.com. She's going to clean up some of these tabs. And I'm going to start a new notebook. And then I'm going to name this one, this is going to be 03 PyTorch computer vision. And I'm going to call mine video.
14:28:17
So just so it has the video tag, because if we go in here, if we go video notebooks of the PyTorch deep learning repo, the video notebooks are stored in here. They've got the underscore video tag. So the video notebooks have all of the code I write exactly in the video. But there are some reference notebooks to go along with it. Let me just write a heading here, PyTorch computer vision. And I'll put a resource here, see reference notebook. Now, of course, this is the one that's the ground truth.
14:28:48
It's got all of the code that we're going to be writing. I'm going to put that in here. Explain with text and images and whatnot. And then finally, as we got see reference online book. And that is at learnpytorch.io at section number three, PyTorch computer vision. I'm going to put that in there. And then I'm going to turn this into markdown with command mm. Beautiful. So let's get started. I'm going to get rid of this, get rid of this. How do we start this off?
14:29:20
Well, I believe there are some computer vision libraries that you should be aware of. Computer vision libraries in PyTorch. So this is just going to be a text based cell. But the first one is torch vision, which is the base domain library for PyTorch computer vision. So if we look up torch vision, what do we find? We have torch vision 0.12. That's the version that torch vision is currently up to at the time of recording this.
14:29:54
So in here, this is very important to get familiar with if you're working on computer vision problems. And of course, in the documentation, this is just another tidbit. We have torch audio for audio problems. We have torch text for text torch vision, which is what we're working on torch rack for recommendation systems torch data for dealing with different data pipelines torch serve, which is for serving PyTorch models and PyTorch on XLA.
14:30:20
So I believe that stands for accelerated linear algebra devices. You don't have to worry about these ones for now. We're focused on torch vision. However, if you would like to learn more about a particular domain, this is where you would go to learn more. So there's a bunch of different stuff that's going on here. Transforming and augmenting images. So fundamentally, computer vision is dealing with things in the form of images. Even a video gets converted to an image.
14:30:45
We have models and pre-trained weights. So as I referenced before, you can use an existing model that works on an existing computer vision problem for your own problem. We're going to cover that in section, I think it's six, for transfer learning. And then we have data sets, which is a bunch of computer vision data sets, utils, operators, a whole bunch of stuff here. So PyTorch is really, really good for computer vision. I mean, look at all the stuff that's going on here.
14:31:09
But that's enough talking about it. Let's just put it in here. Torch vision. This is the main one. I'm not going to link to all of these. All of the links for these, by the way, is in the book version of the course PyTorch Computer Vision. And we have what we're going to cover. And finally, computer vision libraries in PyTorch. Torch vision, data sets, models, transforms, et cetera. But let's just write down the other ones. So we have torch vision, not data sets, something to be aware of.
14:31:38
So get data sets and data loading functions for computer vision here. Then we have torch vision. And from torch vision, models is get pre-trained computer vision. So when I say pre-trained computer vision models, we're going to cover this more in transfer learning, as I said. Pre-trained computer vision models are models that have been already trained on some existing vision data and have trained weights, trained patterns that you can leverage for your own problems,
14:32:11
that you can leverage for your own problems. Then we have torch vision.transforms. And then we have functions for manipulating your vision data, which is, of course, images to be suitable for use with an ML model. So remember, what do we have to do when we have image data or almost any kind of data? For machine learning, we have to prepare it in a way so it can't just be pure images, so that's what transforms help us out with.
14:32:47
Transforms helps to turn our image data into numbers so we can use it with a machine learning model. And then, of course, we have some, these are the torch utils. This is not vision specific, it's entirety of PyTorch specific, and that's data set. So if we wanted to create our own data set with our own custom data, we have the base data set class for PyTorch. And then we have finally torch utils data.
14:33:15
These are just good to be aware of because you'll almost always use some form of data set slash data loader with whatever PyTorch problem you're working on. So this creates a Python iterable over a data set. Wonderful. I think these are most of the libraries that we're going to be using in this section. Let's import some of them, hey, so we can see what's going on. Let's go import PyTorch. Import PyTorch. So import torch. We're also going to get NN, which stands for neural network. What's in NN?
14:33:50
Well, in NN, of course, we have lots of layers, lots of loss functions, a whole bunch of different stuff for building neural networks. We're going to also import torch vision. And then we're going to go from torch vision. Import data sets because we're going to be using data sets later on to get a data set to work with from torch vision. Well, import transforms. You could also go from torch vision dot transforms import to tensor.
14:34:23
This is one of the main ones you'll see for computer vision problems to tensor. You can imagine what it does. But let's have a look. Transforms to tensor. Transforming and augmenting images. So look where we are. We're in pytorch.org slash vision slash stable slash transforms. Over here. So we're in the torch vision section. And we're just looking at transforming and augmenting images. So transforming. What do we have? Transforms are common image transformations of our and the transforms module.
14:34:53
They can be trained together using compose. Beautiful. So if we have two tensor, what does this do? Convert a pill image on NumPy and the array to a tensor. Beautiful. That's what we want to do later on, isn't it? Well, this is kind of me giving you a spoiler is we want to convert our images into tensors so that we can use those with our models.
14:35:14
But there's a whole bunch of different transforms here and actually one of your extra curriculum is to be to read through each of these packages for 10 minutes. So that's about an hour of reading, but it will definitely help you later on if you get familiar with using the pytorch documentation. After all, this course is just a momentum builder. We're going to write heaves of pytorch code. But fundamentally, you'll be teaching yourself a lot of stuff by reading the documentation.
14:35:38
Let's keep going with this. Where were we up to? When we're getting familiar with our data, mapplotlib is going to be fundamental for visualization. Remember, the data explorer's motto, visualize, visualize, visualize, become one with the data. So we're going to import mapplotlib.pyplot as PLT. And then finally, let's check the versions. So print torch.version or underscore, underscore version and print torch vision.
14:36:10
So by the time you watch this, there might be a newer version of each of these modules out. If there's any errors in the code, please let me know. But this is just a bare minimum version that you'll need to complete this section. I believe at the moment, Google Colab is running 1.11 for torch and maybe 1.10. We'll find out in a second. It just connected. So we're importing pytorch. Okay, there we go. So my pytorch version is 1.10 and it's got CUDA available and torch vision is 0.11.
14:36:46
So just make sure if you're running in Google Colab, if you're running this at a later date, you probably have at minimum these versions, you might even have a later version. So these are the minimum versions required for this upcoming section. So we've covered the base computer vision libraries in pytorch. We've got them ready to go. How about in the next video, we cover getting a data set. I'll see you there. Welcome back.
14:37:11
So in the last video, we covered some of the fundamental computer vision libraries in pytorch. The main one being torch vision and then modules that stem off torch vision. And then of course, we've got torch utils dot data dot data set, which is the base data set class for pytorch and data loader, which creates a Python irritable over a data set. So let's begin where most machine learning projects do. And that is getting a data set, getting a data set. I'm going to turn this into markdown.
14:37:38
And the data set that we're going to be used to demonstrating some computer vision techniques is fashion amnest. Which is a take of the data set we'll be using is fashion amnest, which is a take on the original amnest data set, amnest database, which is modified national institute of standards and technology database, which is kind of like the hello world in machine learning and computer vision, which is these are sample images from the amnest test data set,
14:38:11
which are grayscale images of handwritten digits. So this, I believe was originally used for trying to find out if you could use computer vision at a postal service to, I guess, recognize post codes and whatnot. I may be wrong about that, but that's what I know. Yeah 1998. So all the way back at 1998, how cool is that? So this was basically where convolutional neural networks were founded. I'll let you read up on the history of that.
14:38:38
But neural network started to get so good that this data set was quite easy for them to do really well. And that's when fashion amnest came out. So this is a little bit harder if we go into here. This is by Zalando research fashion amnest. And it's of grayscale images of pieces of clothing. So like we saw before the input and output, what we're going to be trying to do is turning these images of clothing into numbers
14:39:06
and then training a computer vision model to recognize what the different styles of clothing are. And here's a dimensionality plot of all the different items of clothing. Visualizing where similar items are grouped together, there's the shoes and whatnot. Is this interactive? Oh no, it's a video. Excuse me. There we go. To serious machine learning researchers. We are talking about replacing amnest. Amnest is too easy. Amnest is overused. Amnest cannot represent modern CV tasks.
14:39:35
So even now fashion amnest I would say has also been pretty much sold, but it's a good way to get started. Now, where could we find such a data set? We could download it from GitHub. But if we come back to the taught division documentation, have a look at data sets. We have a whole bunch of built-in data sets. And remember, this is your extra curricular to read through these for 10 minutes or so each. But we have an example. We could download ImageNet if we want.
14:40:02
We also have some base classes here for custom data sets. We'll see that later on. But if we scroll through, we have image classification data sets. Caltech 101. I didn't even know what all of these are. There's a lot here. CFAR 100. So that's an example of 100 different items. So that would be a 100 class, multi-class classification problem. CFAR 10 is 10 classes. We have amnest. We have fashion amnest. Oh, that's the one we're after.
14:40:29
But this is basically what you would do to download a data set from taughtvision.datasets. You would download the data in some way, shape, or form. And then you would turn it into a data loader. So ImageNet is one of the most popular or is probably the gold standard data set for computer vision evaluation. It's quite a big data set. It's got millions of images. But that's the beauty of taught vision is that it allows us to download example data sets that we can practice on.
14:41:00
I don't even perform research on from a built-in module. So let's now have a look at the fashion amnest data set. How might we get this? So we've got some example code here, or this is the documentation. taughtvision.datasets.fashion amnest. We have to pass in a root. So where do we want to download the data set? We also have to pass in whether we want the training version of the data set or whether we want the testing version of the data set. Do we want to download it? Yes or no?
14:41:27
Should we transform the data in any way shape or form? So we're going to be downloading images through this function call or this class call. Do we want to transform those images in some way? What do we have to do to images before we can use them with a model? We have to turn them into a tensor, so we might look into that in a moment. And target transform is do we want to transform the labels in any way shape or form? So often the data sets that you download from taughtvision.datasets
14:41:54
are pre formatted in a way that they can be quite easily used with PyTorch. But that won't always be the case with your own custom data sets. However, what we're about to cover is just important to get an idea of what the computer vision workflow is. And then later on you can start to customize how you get your data in the right format to be used with the model. Then we have some different parameters here and whatnot. Let's just rather than look at the documentation, if and down, code it out.
14:42:20
So we'll be using fashion MNIST and we'll start by, I'm going to just put this here, from taughtvision.datasets. And we'll put the link there and we'll start by getting the training data. Set up training data. I'm just going to make some code cells here so that I can code in the middle of the screen. Set up training data. Training data equals data sets dot fashion MNIST. Because recall, we've already from taughtvision.
14:42:55
We don't need to import this again, I'm just doing it for demonstration purposes, but from taughtvision import data sets so we can just call data sets dot fashion MNIST. And then we're going to type in root. See how the doc string comes up and tells us what's going on. I personally find this a bit hard to read in Google Colab, so if I'm looking up the documentation, I like to just go into here. But let's code it out. So root is going to be data, so where to download data to.
14:43:25
We'll see what this does in a minute. Then we're going to go train. We want the training version of the data set. So as I said, a lot of the data sets that you find in taughtvision.datasets have been formatted into training data set and testing data set already. So this Boolean tells us do we want the training data set? So if that was false, we would get the testing data set of fashion MNIST. Do we want to download it? Do we want to download? Yes no.
14:43:57
So yes, we do. We're going to set that to true. Now what sort of transform do we want to do? So because we're going to be downloading images and what do we have to do to our images to use them with a machine-loading model, we have to convert them into tensors. So I'm going to pass the transform to tensor, but we could also just go torchvision.transforms.to tensor. That would be the exact same thing as what we just did before. And then the target transform, do we want to transform the labels?
14:44:27
No, we don't. We're going to see how they come, or the target, sorry. High torch, this is another way, another naming convention. Often uses target for the target that you're trying to predict. So using data to predict the target, which is I often use data to predict a label. They're the same thing. So how do we want to transform the data? And how do we want to transform the labels? And then we're going to do the same for the test data. So we're going to go data sets.
14:45:03
You might know what to do here. It's going to be the exact same code as above, except we're going to change one line. We want to store it in data. We want to download the training data set as false because we want the testing version. Do we want to download it? Yes, we do. Do we want to transform it the data? Yes, we do, we want to use to tensor to convert our image data to tensors. And do we want to do a target transform? Well, no, we don't.
14:45:30
We want to keep the label slash the targets as they are. Let's see what happens when we run this. Oh, downloading fashion, Evan is beautiful. So this is going to download all of the labels. What do we have? Train images, train labels, lovely, test images, test labels, beautiful. So that's how quickly we can get a data set by using torch vision data sets. Now, if we have a look over here, we have a data folder because we set the root to be data.
14:45:57
Now, if we look what's inside here, we have fashion MNIST, exactly what we wanted. Then we have the raw, and then we have a whole bunch of files here, which torch vision has converted into data sets for us. So let's get out of that. And this process would be much the same if we used almost any data set in here. They might be slightly different depending on what the documentation says and depending on what the data set is.
14:46:21
But that is how easy torch vision data sets makes it to practice on example computer vision data sets. So let's go back. Let's check out some parameters or some attributes of our data. How many samples do we have? So we'll check the lengths. So we have 60,000 training examples and 10,000 testing examples. So what we're going to be doing is we're going to be building a computer vision model to find patterns in the training data and then use those patterns to predict on the test data.
14:47:01
And so let's see a first training example. See the first training example. So we can just index on the train data. Let's get the zero index and then we're going to have a look at the image and the label. Oh my goodness. A whole bunch of numbers. Now you see what the two tensor has done for us? So we've downloaded some images and thanks to this torch vision transforms to tensor. How would we find the documentation for this? Well, we could go and see what this does transforms to tensor.
14:47:36
We could go to tensor. There we go. What does this do? Convert a pill image. So that's Python image library image on NumPy array to a tensor. This transform does not support torch script. So converts a pill image on NumPy array height with color channels in the range 0 to 255 to a torch float tensor of shape. See here? This is what I was talking about how PyTorch defaults with a lot of transforms to CHW. So color channels first height then width in that range of zero to one.
14:48:07
So typically red, green and blue values are between zero and 255. But neural networks like things between zero and one. And in this case, it is now in the shape of color channels first, then height, then width. However, some other machine learning libraries prefer height, width, then color channels. Just keep that in mind. We're going to see this in practice later on. So we've got an image. We've got a label. Let's check out some more details about it. Remember how we discussed?
14:48:34
Oh, there's our label, by the way. So nine, we can go traindata.classes, find some information about our class names. Class names. Beautiful. So number nine would be 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. So this particular tensor seems to relate to an ankle boot. How would we find that out? Well, one second. I'm just going to show you one more thing, class to IDX. Let's go traindata.class to IDX. What does this give us? Class to IDX.
14:49:10
This is going to give us a dictionary of different labels and their corresponding index. So if our machine learning model predicted nine or class nine, we can convert that to ankle boot using this attribute of the train data. There are more attributes that you can have a look at if you like. You can go traindata.dot, then I just push tab to find out a bunch of different things. You can go data. That'll be the images, and then I believe you can also go targets.
14:49:37
So targets, that's all the labels, which is one big long tensor. Now let's check the shape. Check the shape of our image. So image.shape and label.shape. What are we going to get from that? Oh, label doesn't have a shape. Why is that? Well, because it's only an integer. So oh, beautiful. Look at that. So our image shape is we have a color channel of one. So let me print this out in something prettier, print image shape, which is going to be image shape.
14:50:14
Remember how I said it's very important to be aware of the input and output shapes of your models and your data. It's all part of becoming one with the data. So that is what our image shape is. And then if we go next, this is print image label, which is label, but we'll index on class names for label. And then we'll do that wonderful. So our image shape is currently in the format of color channels height width. We got a bunch of different numbers that's representing our image.
14:50:50
It's black and white. It only has one color channel. Why do you think it only has one color channel? Because it's black and white, so if we jump back into the keynote, fashion, we've already discussed this, grayscale images have one color channel. So that means that for black, the pixel value is zero. And for white, it's some value for whatever color is going on here. So if it's a very high number, say it's one, it's going to be pure white. If it's like 0.001, it might be a faint white pixel.
14:51:20
But if it's exactly zero, it's going to be black. So color images have three color channels for red, green and blue, grayscale have one color channel. But I think we've done enough of visualizing our images as numbers. How about in the next video, we visualize our image as an image? I'll see you there. Welcome back. So in the last video, we checked the input output shapes of our data, and we downloaded the fashion MNIST data set, which is comprised of images or grayscale images of T-shirts,
14:51:54
trousers, pullovers, dress, coat, sandal, shirt, sneaker, bag, ankle boot. Now we want to see if we can build a computer vision model to decipher what's going on in fashion MNIST. So to separate, to classify different items of clothing based on their numerical representation. And part of becoming one with the data is, of course, checking the input output shapes of it. So this is a fashion MNIST data set from Zalando Research. Now if you recall, why did we look at our input and output shapes?
14:52:27
Well, this is what we looked at before. We have 28 by 28 grayscale images that we want to represent as a tensor. We want to use them as input into a machine learning algorithm, typically a computer vision algorithm, such as a CNN. And we want to have some sort of outputs that are formatted in the ideal shape that we'd like. So in our case, we have 10 different types of clothing. So we're going to have an output shape of 10, but our input shape is what?
14:52:54
So by default, PyTorch turns tensors into color channels first. So we have an input shape of none, one, 28, 28. So none is going to be our batch size, which of course we can set that to whatever we'd like. Now input shape format is in NCHW, or in other words, color channels first. But just remember, if you're working with some other machine learning libraries, you may want to use color channels last. So let's have a look at where that might be the case. We're going to visualize our images.
14:53:27
So I make a little heading here, 1.2. Now this is all part of becoming one with the data. In other words, understanding its input and output shapes, how many samples there are, what they look like, visualize, visualize, visualize. Let's import mapplotlib. I'm just going to add a few code cells here, import mapplotlib.pyplot as PLT. Now let's create our image and label is our train data zero, and we're going to print
14:54:03
the image shape so we can understand what inputs are going into our mapplotlib function. And then we're going to go plot.imshow, and we're going to pass in our image and see what happens, because recall what does our image look like, image? Our image is this big tensor of numbers. And we've got an image shape, 128, 128. Now what happens if we call plot.imshow? What happens there? Oh, we get an error in valid shape, 128, 128 for image data.
14:54:40
Now as I said, this is one of the most common errors in machine learning is a shape issue. So the shape of your input tensor doesn't match the expected shape of that tensor. So this is one of those scenarios where our data format, so color channels first, doesn't match up with what mapplotlib is expecting. So mapplotlib expects either just height and width, so no color channel for gray style images, or it also expects the color channels to be last.
14:55:08
So we'll see that later on, but for grayscale, we can get rid of that extra dimension by passing in image.squeeze. So do you recall what squeeze does? It's going to remove that singular dimension. If we have a look at what goes on now, beautiful, we get an ankle boot. Well, that's a very pixelated ankle boot, but we're only dealing with 28 by 28 pixels, so not a very high definition image. Let's add the title to it. We're going to add in the label. Beautiful. So we've got the number nine here.
14:55:43
So where if we go up to here, that's an ankle boot. Now let's plot this in grayscale. How might we do that? We can do the same thing. We can go plotplt.imshow. We're going to pass in image.squeeze. And we're going to change the color map, C map equals gray. So in mapplotlib, if you ever have to change the colors of your plot, you want to look into the C map property or parameter, or sometimes it's also shortened to just C.
14:56:09
But in this case, M show is C map, and we want to plot title, and we're going to pull it in class names and the label integer here. So we have a look at it now. We have an ankle boot, and we can remove the accesses to if we wanted to plot.access, and turn that off. That's going to remove the access. So there we go. That's the type of images that we're dealing with. But that's only a singular image. How about we harness the power of randomness and have a look at some random images from
14:56:38
our data set? So how would we do this? Let's go plot more images. We'll set a random seed. So you and I are both looking at as similar as possible images, 42. Now we'll create a plot by calling plot.figure, and we're going to give it a size. We might create a nine by nine grid. So we want to see nine random images from our data set. So rows, calls, or sorry, maybe we'll do four by four. That'll give us 16. We're going to go four i in range, and we're going to go one to rows times columns plus
14:57:16
one. So we can print i. What's that going to give us? We want to see 16 images. Oh, they're about. So 16 random images, but used with a manual C to 42 of our data set. This is one of my favorite things to do with any type of data set that I'm looking at, whether it be text, image, audio, doesn't matter. I like to randomly have a look at a whole bunch of samples at the start so that I can become one with the data. With that being said, let's use this loop to grab some random indexes.
14:57:50
We can do so using tortures, rand, int, so random integer between zero and length of the training data. This is going to give us a random integer in the range of zero and however many training samples we have, which in our case is what, 60,000 or thereabouts. So we want to create the size of one, and we want to get the item from that so that we have a random index. What is this going to give us? Oh, excuse me, maybe we print that out. There we go. So we have random images.
14:58:21
Now, because we're using manual seed, it will give us the same numbers every time. So we have three, seven, five, four, two, three, seven, five, four, two. And then if we just commented out the random seed, we'll get different numbers every time. But this is just to demonstrate, we'll keep the manual seed there for now. You can comment that out if you want different numbers or different images, different indexes each time.
14:58:45
So we'll create the image and the label by indexing on the training data at the random index that we're generating. And then we'll create our plot. So Fig or we'll add a subplot, Fig add subplot, and we're going to go rows, calls, I. So at the if index, we're going to add a subplot. Remember, we set rows and columns up to here. And then we're going to go PLT dot in show, we're going to show what we're going to show
14:59:13
our image, but we have to squeeze it to get rid of that singular dimension as the color channel. Otherwise, we end up with an issue with map plot lib. We're going to use a color map of gray. So it looks like the image we plotted above. And then for our title, it's going to be our class names indexed with our label. And then we don't want the accesses because that's going to clutter up our plot. Let's see what this looks like. Oh my goodness, look at that. It worked first. Go.
14:59:37
Usually visualizations take a fair bit of trial and error. So we have ankle boots, we have shirts, we have bags, we have ankle boots, sandal, shirt, pull over. Oh, do you notice something about the data set right now, pull over and shirt? To me, they look quite similar. Do you think that will cause an issue later on when our model is trying to predict between a pull over and a shirt? How about if we look at some more images?
15:00:02
We'll get rid of the random seed so we can have a look at different styles. So have a sandal ankle boot coat, t-shirt, top, shirt, oh, is that a little bit confusing that we have a class for t-shirt and top and shirt? Like I'm not sure about you, but what's the difference between a t-shirt and a shirt? This is just something to keep in mind as a t-shirt and top, does that look like it could be maybe even a dress? Like the shape is there. So this is just something to keep in mind going forward.
15:00:34
The chances are if we get confused on our, like you and I looking at our data set, if we get confused about different samples and what they're labeled with, our model might get confused later on. So let's have a look at one more and then we'll go into the next video. So we have sneaker, trouser, shirt, sandal, dress, pull over, bag, bag, t-shirt, oh, that's quite a difficult one. It doesn't look like there's even much going on in that image.
15:01:00
But the whole premise of building machine learning models to do this would be could you write a program that would take in the shapes of these images and figure out, write a rule-based program that would go, hey, if it's looked like a rectangle with a buckle in the middle, it's probably a bag? I mean, you probably could after a while, but I prefer to write machine learning algorithms to figure out patterns and data. So let's start moving towards that.
15:01:26
We're now going to go on figuring out how we can prepare this data to be loaded into a model. I'll see you there. All right, all right, all right. So we've got 60,000 images of clothing that we'd like to build a computer vision model to classify into 10 different classes. And now that we've visualized a fair few of these samples, do you think that we could model these with just linear lines, so straight lines, or do you think we'll need a model with nonlinearity? So I'm going to write that down.
15:01:58
So do you think these items of clothing images could be modeled with pure linear lines, or do you think we'll need nonlinearity? Don't have to answer that now. We could test that out later on. You might want to skip ahead and try to build a model yourself with linear lines or nonlinearities. We've covered linear lines and nonlinearities before, but let's now start to prepare our data even further to prepare data loader. So right now, our data is in the form of PyTorch data sets.
15:02:46
So let's have a look at it. Same data. There we go. So we have data set, which is of fashion MNIST. And then if we go test data, we see a similar thing except we have a different number of data points. We have the same transform on each, we've turned them into tenses. So we want to convert them from a data set, which is a collection of all of our data, into a data loader. Paul, that a data loader, turns our data set into a Python iterable. So I'm going to turn this into Markdown, beautiful.
15:03:26
More specifically, specific Galilee, can I spell right? I don't know, we want to just code right, we're not here to learn spelling. We want to turn our data into batches, or mini batches. Why would we do this? Well, we may get away with it by building a model to look at all 60,000 samples of our current data set, because it's quite small. It's only comprised of images of 28 by 28 pixels. And when I say quite small, yes, 60,000 images is actually quite small for a deep learning scale data set.
15:04:08
Modern data sets could be in the millions of images. But if our computer hardware was able to look at 60,000 samples of 28 by 28 at one time, it would need a fair bit of memory. So we have RAM space up here, we have GPU memory, we have compute memory. But chances are that it might not be able to store millions of images in memory. So what you do is you break a data set from say 60,000 into groups of batches or mini batches. So we've seen batch size before, why would we do this?
15:04:45
Well, one, it is more computationally efficient, as in your computing hardware may not be able to look store in memory at 60,000 images in one hit. So we break it down to 32 images at a time. This would be batch size of 32. Now again, 32 is a number that you can change. 32 is just a common batch size that you'll see with many beginner style problems. As you go on, you'll see different batch sizes. This is just to exemplify the concept of mini batches, which is very common in deep learning.
15:05:30
And why else would we do this? The second point or the second main point is it gives our neural network more chances to update its gradients per epoch. So what I mean by this, this will make more sense when we write a training loop. But if we were to just look at 60,000 images at one time, we would per epoch. So per iteration through the data, we would only get one update per epoch across our entire data set.
15:06:00
Whereas if we look at 32 images at a time, our neural network updates its internal states, its weights, every 32 images, thanks to the optimizer. This will make a lot more sense once we write our training loop. But these are the two of the main reasons for turning our data into mini batches in the form of a data loader. Now if you'd like to learn more about the theory behind this, I would highly recommend looking up Andrew Org mini batches. There's a great lecture on that.
15:06:28
So yeah, large-scale machine learning, mini batch gradient descent, mini batch gradient descent. Yeah, that's what it's called mini batch gradient descent. If you look up some results on that, you'll find a whole bunch of stuff. I might just link this one, I'm going to pause that, I'm going to link this in there. So for more on mini batches, see here. Now to see this visually, I've got a slide prepared for this. So this is what we're going to be working towards.
15:07:02
There's our input and output shapes. We want to create batch size of 32 across all of our 60,000 training images. And we're actually going to do the same for our testing images, but we only have 10,000 testing images. So this is what our data set's going to look like, batched. So we're going to write some code, namely using the data loader from torch.util.data. We're going to pass it a data set, which is our train data.
15:07:26
We're going to give it a batch size, which we can define as whatever we want. For us, we're going to use 32 to begin with. And we're going to set shuffle equals true if we're using the training data. Why would we set shuffle equals true? Well, in case our data set for some reason has order, say we had all of the pants images in a row, we had all of the T-shirt images in a row, we had all the sandal images in a row. We don't want our neural network to necessarily remember the order of our data.
15:07:51
We just want it to remember individual patterns between different classes. So we shuffle up the data, we mix it, we mix it up. And then it looks something like this. So we might have batch number zero, and then we have 32 samples. Now I ran out of space when I was creating these, but we got, that was fun up to 32. So this is setting batch size equals 32. So we look at 32 samples per batch. We mix all the samples up, and we go batch, batch, batch, batch, batch, and we'll have,
15:08:22
however many batches we have, we'll have number of samples divided by the batch size. So 60,000 divided by 32, what's that, 1800 or something like that? So this is what we're going to be working towards. I did want to write some code in this video, but I think to save it getting too long, we're going to write this code in the next video. If you would like to give this a go on your own, here's most of the code we have to do. So there's the train data loader, do the same for the test data loader.
15:08:46
And I'll see you in the next video, and we're going to batchify our fashion MNIST data set. Welcome back. In the last video, we had a brief overview of the concept of mini batches. And so rather than our computer looking at 60,000 images in one hit, we break things down. We turn it into batches of 32. Again, the batch size will vary depending on what problem you're working on. But 32 is quite a good value to start with and try out.
15:09:15
And we do this for two main reasons, if we jump back to the code, why would we do this? It is more computationally efficient. So if we have a GPU with, say, 10 gigabytes of memory, it might not be able to store all 60,000 images in one hit. In our data set, because it's quite small, it may be hour or two, but it's better practice for later on to turn things into mini batches. And it also gives our neural network more chances to update its gradients per epoch,
15:09:42
which will make a lot more sense once we write our training loop. But for now, we've spoken enough about the theory. Let's write some code to do so. So I'm going to import data loader from torch dot utils dot data, import data loader. And this principle, by the way, preparing a data loader goes the same for not only images, but for text, for audio, whatever sort of data you're working with, mini batches will
15:10:08
follow you along or batches of data will follow you along throughout a lot of different deep learning problems. So set up the batch size hyper parameter. Remember, a hyper parameter is a value that you can set yourself. So batch size equals 32. And it's practice. You might see it typed as capitals. You won't always see it, but you'll see it quite often a hyper parameter typed as capitals. And then we're going to turn data sets into iterables. So batches.
15:10:40
So we're going to create a train data loader here of our fashion MNIST data set. We're going to use data loader. We're going to see what the doc string is. Or actually, let's look at the documentation torch data loader. This is some extra curriculum for you too, by the way, is to read this data page torch utils not data because no matter what problem you're going with with deep learning or pytorch, you're going to be working with data. So spend 10 minutes just reading through here.
15:11:09
I think I might have already assigned this, but this is just so important that it's worth going through again. Read through all of this. Even if you don't understand all of it, what's going on, it's just it helps you know where to look for certain things. So what does it take? Data loader takes a data set. We need to set the batch size to something is the default of one. That means that it would create a batch of one image at a time in our case. Do we want to shuffle it?
15:11:33
Do we want to use a specific sampler? There's a few more things going on. Number of workers. Number of workers stands for how many cores on our machine do we want to use to load data? Generally the higher the better for this one, but we're going to keep most of these as the default because most of them are set to pretty good values to begin with. I'll let you read more into the other parameters here. We're going to focus on the first three data set batch size and shuffle true or false.
15:12:00
Let's see what we can do. So data set equals our train data, which is 60,000 fashion MNIST. And then we have a batch size, which we're going to set to our batch size hyper parameter. So we're going to have a batch size of 32. And then finally, do we want to shuffle the training data? Yes, we do. And then we're going to do the same thing for the test data loader, except we're not going to shuffle the test data.
15:12:25
Now, you can shuffle the test data if you want, but in my practice, it's actually easier to evaluate different models when the test data isn't shuffled. So you shuffle the training data to remove order. And so your model doesn't learn order. But for evaluation purposes, it's generally good to have your test data in the same order because our model will never actually see the test data set during training. We're just using it for evaluation.
15:12:53
So the order doesn't really matter to the test data loader. It's just easier if we don't shuffle it, because then if we evaluate it multiple times, it's not been shuffled every single time. So let's run that. And then we're going to check it out, our train data loader and our test data loader. Beautiful. Instances of torch utils data, data loader, data loader. And now let's check out what we've created, hey, I always like to print different attributes
15:13:25
of whatever we make, check out what we've created. This is all part of becoming one with the data. So print F, I'm going to go data loaders, and then pass in, this is just going to output basically the exact same as what we've got above. This data loader. And we can also see what attributes we can get from each of these by going train data loader. I don't need caps lock there, train data loader, full stop. And then we can go tab. We've got a whole bunch of different attributes.
15:13:57
We've got a batch size. We've got our data set. Do we want to drop the last as in if our batch size overlapped with our 60,000 samples? Do we want to get rid of the last batch? Say for example, the last batch only had 10 samples. Do we want to just drop that? Do we want to pin the memory that's going to help later on if we wanted to load our data faster? A whole bunch of different stuff here. If you'd like to research more, you can find all the stuff about what's going on here in
15:14:22
the documentation. But let's just keep pushing forward. What else do we want to know? So let's find the length of the train data loader. We will go length train data loader. So this is going to tell us how many batches there are, batches of, which of course is batch size. And we want print length of test data loader. We want length test data loader batches of batch size dot dot dot. So let's find out some information. What do we have? Oh, there we go.
15:15:03
So just we're seeing what we saw before with this one. But this is more interesting here. Length of train data loader. Yeah, we have about 1,875 batches of 32. So if we do 60,000 training samples divided by 32, yeah, it comes out to 1,875. And if we did the same with 10,000 for testing samples of 32, it comes out at 313. This gets rounded up. So this is what I meant, that the last batch will have maybe not 32 because 32 doesn't divide evenly into 10,000, but that's okay.
15:15:36
And so this means that our model is going to look at 1,875 individual batches of 32 images, rather than just one big batch of 60,000 images. Now of course, the number of batches we have will change if we change the batch size. So we have 469 batches of 128. And if we reduce this down to one, what do we get? We have a batch per sample. So 60,000 batches of 1, 10,000 batches of 1, we're going to stick with 32. But now let's visualize. So we've got them in train data loader.
15:16:12
How would we visualize a batch or a single image from a batch? So let's show a sample. I'll show you how you can interact with a data loader. We're going to use randomness as well. So we'll set a manual seed and then we'll get a random index, random idx equals torch rand int. We're going to go from zero to length of train features batch. Oh, where did I get that from? Excuse me. Getting ahead of myself here. I want to check out what's inside the training data loader.
15:16:48
We'll check out what's inside the training data loader because the test data load is going to be similar. So we want the train features batch. So I say features as in the images themselves and the train labels batch is going to be the labels of our data set or the targets in pytorch terminology. So next idar data loader. So because our data loader has 1875 batches of 32, we're going to turn it into an iterable
15:17:16
with ita and we're going to get the next batch with next and then we can go here train features batch.shape and we'll get train labels batch.shape. What do you think this is going to give us? Well, there we go. Look at that. So we have a tensor. Each batch we have 32 samples. So this is batch size and this is color channels and this is height and this is width. And then we have 32 labels associated with the 32 samples.
15:17:49
Now where have we seen this before, if we go back through our keynote input and output shapes. So we have shape equals 32, 28, 28, 1. So this is color channels last, but ours is currently in color channels first. Now again, I sound like a broken record here, but these will vary depending on the problem you're working with. If we had larger images, what would change or the height and width dimensions would change.
15:18:17
If we had color images, the color dimension would change, but the premise is still the same. We're turning our data into batches so that we can pass that to a model. Let's come back. Let's keep going with our visualization. So we want to visualize one of the random samples from a batch and then we're going to go image label equals train features batch and we're going to get the random IDX from that and we'll get the train labels batch and we'll get the random IDX from that.
15:18:50
So we're matching up on the, we've got one batch here, train features batch, train labels batch and we're just getting the image and the label at a random index within that batch. So excuse me, I need to set this equal there. And then we're going to go PLT dot in show, what are we going to show? We're going to show the image but we're going to have to squeeze it to remove that singular dimension and then we'll set the C map equal to gray and then we'll go PLT dot title, we'll
15:19:23
set the title which is going to be the class names indexed by the label integer and then we can turn off the accesses. You can use off here or you can use false, depends on what you'd like to use. Let's print out the image size because you can never know enough about your data and then print, let's also get the label, label and label shape or label size. Our label will be just a single integer so it might not have a shape but that's okay. Let's have a look. Oh bag.
15:20:04
See, look, that's quite hard to understand. I wouldn't be able to detect that that's a bag. Can you tell me that you could write a program to understand that? That just looks like a warped rectangle to me. But if we had to look at another one, we'll get another random, oh, we've got a random seed so it's going to produce the same image each time. So we have a shirt, okay, a shirt. So we see the image size there, 128, 28.
15:20:28
Now, recall that the image size is, it's a single image so it doesn't have a batch dimension. So this is just color channels height width. We'll go again, label four, which is a coat and we could keep doing this to become more and more familiar with our data. But these are all from this particular batch that we created here, coat and we'll do one more, another coat. We'll do one more just to make sure it's not a coat. There we go. We've got a bag. Beautiful.
15:20:58
So we've now turned our data into data loaders. So we could use these to pass them into a model, but we don't have a model. So I think it's time in the next video, we start to build model zero. We start to build a baseline. I'll see you in the next video. Welcome back. So in the last video, we got our data sets or our data set into data loaders. So now we have 1,875 batches of 32 images off of the training data set rather than 60,000 in a one big data set.
15:21:33
And we have 13 or 313 batches of 32 for the test data set. Then we learned how to visualize it from a batch. And we saw that we have still the same image size, one color channel, 28, 28. All we've done is we've turned them into batches so that we can pass them to our model. And speaking of model, let's have a look at our workflow. Where are we up to? Well, we've got our data ready. We've turned it into tensors through a combination of torch vision transforms, torch utils data dot data set.
15:22:04
We didn't have to use that one because torch vision dot data sets did it for us with the fashion MNIST data set, but we did use that one. We did torch utils dot data, the data loader to turn our data sets into data loaders. Now we're up to building or picking a pre-trained model to suit your problem. So let's start simply. Let's build a baseline model. And this is very exciting because we're going to build our first model, our first computer
15:22:29
vision model, albeit a baseline, but that's an important step. So I'm just going to write down here. When starting to build a series of machine learning modeling experiments, it's best practice to start with a baseline model. I'm going to turn this into markdown. A baseline model. So a baseline model is a simple model. You will try and improve upon with subsequent models, models slash experiments. So you start simply, in other words, start simply and add complexity when necessary because
15:23:22
neural networks are pretty powerful, right? And so they have a tendency to almost do too well on our data set. That's a concept known as overfitting, which we'll cover a little bit more later. But we built a simple model to begin with, a baseline. And then our whole goal will be to run experiments, according to the workflow, improve through experimentation. Again, this is just a guide. It's not set in stone, but this is the general pattern of how things go.
15:23:47
Get data ready, build a model, fit the model, evaluate, improve the model. So the first model that we build is generally a baseline. And then later on, we want to improve through experimentation. So let's start building a baseline. But I'm going to introduce to you a new layer that we haven't seen before. That is creating a flatten layer. Now what is a flatten layer? Well, this is best seen when we code it out. So let's create a flatten model, which is just going to be nn.flatten.
15:24:15
And where could we find the documentation for this? We go nn flatten, flatten in pytorch, what does it do? Flattens a continuous range of dims into a tensor, for use with sequential. So there's an example there, but I'd rather, if and doubt, code it out. So we'll create the flatten layer. And of course, all nn.flatten or nn.modules could be used as a model on their own. So we're going to get a single sample. So x equals train features batch. Let's get the first one, zero.
15:24:54
What does this look like? So it's a tensor, x, maybe we get the shape of it as well, x shape. What do we get? There we go. So that's the shape of x. Keep that in mind when we pass it through the flatten layer. Do you have an inkling of what flatten might do? So our shape to begin with is what, 128, 28. Now let's flatten the sample. So output equals, we're going to pass it to the flatten model, x. So this is going to perform the forward pass internally on the flatten layer.
15:25:32
So perform forward pass. Now let's print out what happened. Print, shape before flattening equals x dot shape. And we're going to print shape after flattening equals output dot shape. So we're just taking the output of the flatten model and printing its shape here. Oh, do you notice what happened? Well we've gone from 128, 28 to 1784. Wow what does the output look like? Output. Oh, the values are now in all one big vector and if we squeeze that we can remove the extra dimension.
15:26:24
So we've got one big vector of values. Now where did this number come from? Well, if we take this and this is what shape is it? We've got color channels. We've got height. We've got width and now we've flattened it to be color channels, height, width. So we've got one big feature vector because 28 by 28 equals what? We've got one value per pixel, 784. One value per pixel in our output vector. Now where did we see this before?
15:27:06
If we go back to our keynote, if we have a look at Tesla's takes eight cameras and then it turns it into a three dimensional vector space, vector space. So that's what we're trying to do here. We're trying to encode whatever data we're working with in Tesla's case. They have eight cameras. Now theirs has more dimensions than ours because they have the time aspect because they're dealing with video and they have multiple different camera angles. We're just dealing with a single image here.
15:27:32
But regardless, the concept is the same. We're trying to condense information down into a single vector space. And so if we come back to here, why might we do this? Well, it's because we're going to build a baseline model and we're going to use a linear layer as the baseline model. And the linear layer can't handle multi dimensional data like this. We want it to have a single vector as input. Now this will make a lot more sense after we've coded up our model.
15:28:00
Let's do that from torch import and then we're going to go class, fashion, amnest, model V zero. We're going to inherit from an end dot module. And inside here, we're going to have an init function in the constructor. We're going to pass in self. We're going to have an input shape, which we'll use a type hint, which will take an integer because remember, input shape is very important for machine learning models.
15:28:31
We're going to define a number of hidden units, which will also be an integer, and then we're going to define our output shape, which will be what do you think our output shape will be? How many classes are we dealing with? We're dealing with 10 different classes. So our output shape will be, I'll save that for later on. I'll let you guess for now, or you might already know, we're going to initialize it. And then we're going to create our layer stack.
15:28:56
self.layer stack equals nn.sequential, recall that sequential, whatever you put inside sequential, if data goes through sequential, it's going to go through it layer by layer. So let's create our first layer, which is going to be nn.flatten. So that means anything that comes into this first layer, what's going to happen to it? It's going to flatten its external dimensions here. So it's going to flatten these into something like this. So we're going to flatten it first, flatten our data.
15:29:25
Then we're going to pass in our linear layer. And we're going to have how many n features this is going to be input shape, because we're going to define our input shape here. And then we're going to go out features, equals hidden units. And then we're going to create another linear layer here. And we're going to set up n features, equals hidden units. Why are we doing this? And then out features equals output shape. Why are we putting the same out features here as the n features here?
15:29:58
Well, because subsequent layers, the input of this layer here, its input shape has to line up with the output shape of this layer here. Hence why we use out features as hidden units for the output of this nn.linear layer. And then we use n features as hidden units for the input value of this hidden layer here. So let's keep going. Let's go def. We'll create the forward pass here, because if we subclass nn.module, we have to override the forward method. The forward method is going to define what?
15:30:33
It's going to define the forward computation of our model. So we're just going to return self.layer stack of x. So our model is going to take some input, x, which could be here, x. In our case, it's going to be a batch at a time, and then it's going to pass each sample through the flatten layer. It's going to pass the output of the flatten layer to this first linear layer, and it's going to pass the output of this linear layer to this linear layer. So that's it.
15:31:05
Our model is just two linear layers with a flatten layer. The flatten layer has no learnable parameters. Only these two do. And we have no nonlinearities. So do you think this will work? Does our data set need nonlinearities? Well, we can find out once we fit our model to the data, but let's set up an instance of our model. So torch dot manual seed. Let's go set up model with input parameters. So we have model zero equals fashion MNIST model, which is just the same class that we wrote above.
15:31:47
And here's where we're going to define the input shape equals 784. Where will I get that from? Well, that's here. That's 28 by 28. So the output of flatten needs to be the input shape here. So we could put 28 by 28 there, or we're just going to put 784 and then write a comment here. This is 28 by 28. Now if we go, I wonder if nn.linear will tell us, nn.linear will tell us what it expects as in features. Size of each input sample, shape, where star means any number of dimensions, including
15:32:32
none in features, linear weight, well, let's figure it out. Let's see what happens if in doubt coded out, hey, we'll see what we can do. In units equals, let's go with 10 to begin with. How many units in the hidden layer? And then the output shape is going to be what? Output shape is length of class names, which will be 1 for every class. Beautiful. And now let's go model zero. We're going to keep it on the CPU to begin with.
15:33:11
We could write device-agnostic code, but to begin, we're going to send it to the CPU. I might just put that up here, actually, to CPU. And then let's have a look at model zero. Wonderful. So we can try to do a dummy forward pass and see what happens. So let's create dummy x equals torch, rand, we'll create it as the same size of image. Just a singular image. So this is going to be a batch of one, color channel one, height 28, height 28. And we're going to go model zero and pass through dummy x.
15:33:50
So this is going to send dummy x through the forward method. Let's see what happens. Okay wonderful. So we get an output of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 logits. Beautiful. That's exactly what we want. We have one logit value per class that we have. Now what would happen if we got rid of flatten? Then we ran this, ran this, ran this. What do we get? Oh, mat one and mat two shapes cannot be multiplied. So we have 28 by 28 and 7. Okay, what happens if we change our input shape to 28?
15:34:34
We're getting shape mismatches here. What happens here? Oh, okay, we get an interesting output, but this is still not the right shape, is it? So that's where the flatten layer comes in. What is the shape of this? Oh, we get 1, 1, 28, 10. Oh, so that's why we put in flatten so that it combines it into a vector. So we get rid of this, see if we just leave it in this shape? We get 28 different samples of 10, which is not what we want.
15:35:05
We want to compress our image into a singular vector and pass it in. So let's reinstanceuate the flatten layer and let's make sure we've got the right input shape here, 28 by 28, and let's pass it through, torch size 110. That's exactly what we want, 1 logit per class. So this could be a bit fiddly when you first start, but it's also a lot of fun once you get it to work. And so just keep that in mind, I showed you what it looks like when you have an error.
15:35:32
One of the biggest errors that you're going to face in machine learning is different tensor shape mismatches. So just keep in mind the data that you're working with and then have a look at the documentation for what input shape certain layers expect. So with that being said, I think it's now time that we start moving towards training our model. I'll see you in the next video. Welcome back. In the last video, we created model zero, which is going to be our baseline model for
15:36:02
our computer vision problem of detecting different types of clothing in 28 by 28 gray scale images. And we also learned the concept of making sure our or we rehashed on the concept of making sure our input and output shapes line up with where they need to be. We also did a dummy forward pass with some dummy data. This is a great way to troubleshoot to see if your model shapes are correct. If they come out correctly and if the inputs are lining up with where they need to be.
15:36:32
And just to rehash on what our model is going to be or what's inside our model, if we check model zero state dict, what we see here is that our first layer has a weight tensor. It also has a bias and our next layer has a weight tensor and it also has a bias. So these are of course initialized with random values, but the whole premise of deep learning and machine learning is to pass data through our model and use our optimizer to update
15:37:02
these random values to better represent the features in our data. And I keep saying features, but I just want to rehash on that before we move on to the next thing. Featuring data could be almost anything. So for example, the feature of this bag could be that it's got a rounded handle at the top. It has a edge over here. It has an edge over there. Now, we aren't going to tell our model what features to learn about the data.
15:37:25
The whole premise of it is to, or the whole fun, the whole magic behind machine learning is that it figures out what features to learn. And so that is what the weights and bias matrices or tensors will represent is different features in our images. And there could be many because we have 60,000 images of 10 classes. So let's keep pushing forward. It's now time to set up a loss function and an optimizer. Speaking of optimizers, so 3.1 set up loss optimizer and evaluation metrics.
15:37:58
Now recall in notebook two, I'm going to turn this into markdown. We created, oh, I don't need an emoji there. So this is, by the way, we're just moving through this workflow. We've got our data ready into tensors. We've built a baseline model. It's now time to pick a loss function and an optimizer. So we go back to Google Chrome. That's right here. Loss function. What's our loss function going to be? Since we're working with multi-class data, our loss function will be NN dot cross entropy loss.
15:38:37
And our optimizer, we've got a few options here with the optimizer, but we've had practice in the past with SGD, which stands for stochastic gradient descent and the atom optimizer. So our optimizer, let's just stick with SGD, which is kind of the entry level optimizer torch opt in SGD for stochastic gradient descent. And finally, our evaluation metric, since we're working on a classification problem, let's use accuracy as our evaluation metric.
15:39:25
So recall that accuracy is a classification evaluation metric. Now, where can we find this? Well, if we go into learnpytorch.io, this is the beauty of having online reference material. In here, neural network classification with PyTorch, in this notebook, section 02, we created, do we have different classification methods? Yes, we did. So we've got a whole bunch of different options here for classification evaluation metrics. We've got accuracy, precision, recall, F1 score, a confusion matrix.
15:39:56
Now we have some code that we could use. If we wanted to use torch metrics for accuracy, we could. And torch metrics is a beautiful library that has a lot of evaluation. Oh, it doesn't exist. What happened to torch metrics? Maybe I need to fix that. Link. Torch metrics has a whole bunch of different PyTorch metrics. So very useful library. But we also coded a function in here, which is accuracy FN. So we could copy this, straight into our notebook here.
15:40:35
Or I've also, if we go to the PyTorch deep learning GitHub, I'll just bring it over here. I've also put it in helper functions.py. And this is a script of common functions that we've used throughout the course, including if we find accuracy function here. Calculate accuracy. Now, how would we get this helper functions file, this Python file, into our notebook? One way is to just copy the code itself, straight here. But let's import it as a Python script.
15:41:04
So import request, and we're going to go from pathlib import path. So we want to download, and this is actually what you're going to see, very common practice in larger Python projects, especially deep learning and machine learning projects, is different functionality split up in different Python files. And that way, you don't have to keep rewriting the same code over and over again. Like you know how we've written a training and testing loop a fair few times?
15:41:30
Well, if we've written it once and it works, we might want to save that to a.py file so we can import it later on. So let's now write some code to import this helper functions.py file into our notebook here. So download helper functions from learn pytorch repo. So we're going to check if our helper functions.py, if this already exists, we don't want to download it. So we'll print helper functions.py already exists, skipping download, skipping download .dot. And we're going to go else here.
15:42:11
If it doesn't exist, so we're going to download it, downloading helper functions.py. And we're going to create a request here with the request library equals request.get. Now here's where we have to pass in the URL of this file. It's not this URL here. When dealing with GitHub, to get the actual URL to the files, many files, you have to click the raw button. So I'll just go back and show you, click raw here. And we're going to copy this raw URL. See how it's just text here?
15:42:47
This is what we want to download into our co-lab notebook. And we're going to write it in there, request equals request.get. And we're going to go with open, and here's where we're going to save our helper functions .py. We're going to write binary as file, F is for file. We're going to go F.write, request.content. So what this is saying is Python is going to create a file called helper functions.py and give it write binary permissions as F, F is for file, short for file.
15:43:22
And then we're going to say F.write, request, get that information from helper functions .py here, and write your content to this file here. So let's give that a shot. Beautiful. So downloading helper functions.py, let's have a look in here. Do we have helper functions.py? Yes, we do. Wonderful. We can import our accuracy function. Where is it? There we go. Import accuracy function. So this is very common practice when writing lots of Python code is to put helper functions into.py scripts.
15:44:04
So let's import the accuracy metric. Accuracy metric from helper functions. Of course, we could have used torch metrics as well. That's another perfectly valid option, but I just thought I'd show you what it's like to import your own helper function script. Of course, you can customize helper functions.py to have whatever you want in there. So see this? We've got from helper functions, import accuracy function. What's this saying? Could not be resolved. Is this going to work? It did.
15:44:38
And where you can go accuracy function, do we get a doc string? Hmm. Seems like colab isn't picking things up, but that's all right. It looks like it still worked. We'll find out later on if it actually works when we train our model. So set up loss function and optimizer. So I'm going to set up the loss function equals nn dot cross entropy loss. And I'm going to set up the optimizer here as we discussed before as torch dot opt-in dot SGD for stochastic gradient descent.
15:45:13
The parameters I want to optimize are the parameters from model zero, our baseline model, which we had a look at before, which are all these random numbers. We'd like our optimizer to tweak them in some way, shape, or form to better represent our data. And then I'm going to set the learning rate here. How much should they be tweaked each epoch? I'm going to set it to 0.1. Nice and high because our data set is quite simple. It's 28 by 28 images. There are 60,000 of them.
15:45:39
But again, if this doesn't work, we can always adjust this and experiment, experiment, experiment. So let's run that. We've got a loss function. Is this going to give me a doc string? There we go. So calculates accuracy between truth and predictions. Now, where does this doc string come from? Well, let's have a look, hope of functions. That's what we wrote before. Good on us for writing good doc strings, accuracy function.
15:46:06
Well, we're going to test all these out in the next video when we write a training loop. So, oh, actually, I think we might do one more function before we write a training loop. How about we create a function to time our experiments? Yeah, let's give that a go in the next video. I'll see you there. Welcome back. In the last video, we downloaded our helper functions.py script and imported our accuracy function that we made in notebook two.
15:46:35
But we could really beef this up, our helper functions.py file. We could put a lot of different helper functions in there and import them so we didn't have to rewrite them. That's just something to keep in mind for later on. But now, let's create a function to time our experiments. So creating a function to time our experiments. So one of the things about machine learning is that it's very experimental. You've probably gathered that so far. So let's write here.
15:47:04
So machine learning is very experimental. Two of the main things you'll often want to track are, one, your model's performance such as its loss and accuracy values, et cetera. And two, how fast it runs. So usually you want a higher performance and a fast model, that's the ideal scenario. However, you could imagine that if you increase your model's performance, you might have a bigger neural network. It might have more layers. It might have more hidden units.
15:47:45
It might degrade how fast it runs because you're simply making more calculations. So there's often a trade-off between these two. And how fast it runs will really be important if you're running a model, say, on the internet or say on a dedicated GPU or say on a mobile device. So these are two things to really keep in mind. So because we're tracking our model's performance with our loss value and our accuracy function, let's now write some code to check how fast it runs.
15:48:14
And I did on purpose above, I kept our model on the CPU. So we're also going to compare later on how fast our model runs on the CPU versus how fast it runs on the GPU. So that's something that's coming up. Let's write a function here. We're going to use the time module from Python. So from time it, import the default timer, as I'm going to call it timer. So if we go Python default timer, do we get the documentation for, here we go, time it. So do we have default timer, wonderful.
15:48:49
So the default timer, which is always time.perf counter, you can read more about Python timing functions in here. But this is essentially just going to say, hey, this is the exact time that our code started. And then we're going to create another stop for when our code stopped. And then we're going to compare the start and stop times. And that's going to basically be how long our model took to train. So we're going to go def print train time. This is just going to be a display function.
15:49:18
So start, we're going to get the float type hint, by the way, start an end time. So the essence of this function will be to compare start and end time. And we're going to set the torch or the device here, we'll pass this in as torch dot device. And we're going to set that default to none, because we want to compare how fast our model runs on different devices. So I'm just going to write a little doc string here, prints, difference between start and end time.
15:49:51
And then of course, we could add more there for the arguments, but that's a quick one liner. Tell us what our function does. So total time equals end minus start. And then print, we're going to write here train time on, whichever device we're using might be CPU, might be GPU. Total time equals, we'll go to three and we'll say seconds, three decimal places that is and return total time. Beautiful. So for example, we could do start time equals timer, and then end time equals timer.
15:50:32
And then we can put in here some code between those two. And then if we go print train, oh, maybe we need a timer like this, we'll find out if and out code it out, you know, we'll see if it works. Start time and end equals end time and device equals. We're running on the CPU right now, CPU, let's see if this works, wonderful. So it's a very small number here. So train time on CPU, very small number, because the start time is basically on this
15:51:13
exact line, comment basically it takes no time to run, then end time is on here, we get 3.304 times 10 to the power of negative five. So quite a small number, but if we put some modeling code in here, it's going to measure the start time of this cell, it's going to model our code in there, then we have the end time, and then we find out how long our model took the train. So with that being said, I think we've got all of the pieces of the puzzle for creating some training and testing functions.
15:51:47
So we've got a loss function, we've got an optimizer, we've got a valuation metric, we've got a timing function, we've got a model, we've got some data. How about we train our first baseline computer vision model in the next video? I'll see you there. Good morning. Well might not be morning wherever you are in the world. It's nice and early here, I'm up recording some videos, because we have a lot of momentum going with this, but look at this, I took a little break last night, I have a runtime
15:52:14
disconnected, but this is just what's going to happen if you're using Google Colab. Since I use Google Colab Pro, completely unnecessary for the course, but I just found it worth it for how much I use Google Colab, I get longer idle timeouts, so that means that my Colab notebook will stay persistent for a longer time. But of course overnight it's going to disconnect, so I click reconnect, and then if I want to
15:52:38
get back to wherever we were, because we downloaded some data from torchvision.datasets, I have to rerun all of these cells. So a nice shortcut, we might have seen this before, is to just come down to where we were, and if all the code above works, oh there we go, I wrote myself some notes of where we're up to. Let's go run before, so this is just going to run all the cells above, and we're up to here, 3.3 creating a training loop, and training a model on batches of data.
15:53:11
So that's going to be a little bit interesting, and I wrote myself another reminder here, this is a little bit of behind the scenes, the optimise will update a model's parameters once per batch rather than once per epoch. So let's hold myself to that note, and make sure I let you know. So we're going to make another title here. Let's go creating a training loop, and training a model on batches of data. So something a little bit different to what we may have seen before if we haven't created
15:53:44
batches of data using data loader, and recall that just up above here, we've got something like 1800 there, there we go. So we've split our data into batches, rather than our model looking at 60,000 images of fashion MNIST data at one time, it's going to look at 1875 batches of 32, so 32 images at the time, of the training data set, and 313 batches of 32 of the test data set. So let's go to training loop and train our first model.
15:54:19
So I'm going to write out a few steps actually, because we have to do a little bit differently to what we've done before. So one, we want to loop through epochs, so a number of epochs. Loop through training batches, and by the way, you might be able to hear some birds singing, the sun is about to rise, I hope you enjoy them as much as I do. So we're going to perform training steps, and we're going to calculate calculate the train loss per batch.
15:54:49
So this is going to be one of the differences between our previous training loops. And this is going to, after number two, we're going to loop through the testing batches. So we'll train and evaluate our model at the same step, or same loop. And we're going to perform testing steps. And then we're going to calculate the test loss per batch as well, per batch. Wonderful, four, we're going to, of course, print out what's happening.
15:55:23
You may have seen the unofficial PyTorch optimization loop theme song. And we're going to time it all for fun, of course, because that's what our timing function is for. So let's get started. There's a fair few steps here, but nothing that we can't handle. And remember the motto, if and out, code it out. Well, there's another one, if and out, run the code, but we haven't written any code to run just yet. So we're going to import TQDM for a progress bar.
15:55:52
If you haven't seen TQDM before, it's a very good Python progress bar that you can add with a few lines of code. So this is just the GitHub. It's open source software, one of my favorite pieces of software, and it's going to give us a progress bar to let us know how many epochs our training loop has gone through. It doesn't have much overhead, but if you want to learn more about it, please refer to the TQDM GitHub.
15:56:17
However, the beautiful thing is that Google CoLab has TQDM built in because it's so good and so popular. So we're going to import from TQDM.auto. So there's a few different types of TQDM progress bars.auto is just going to recognize what compute environment we're using. And it's going to give us the best type of progress bar for what we're doing. So for example, Google CoLab is running a Jupyter Notebook behind the scenes.
15:56:45
So the progress bar for Jupyter Notebooks is a little bit different to Python scripts. So now let's set the seed and start the timer. We want to write all of our training loop in this single cell here. And then once it starts, once we run this cell, we want the timer to start so that we can time how long the entire cell takes to run. So we'll go train time start on CPU equals, we set up our timer before, beautiful. Now we're going to set the number of epochs.
15:57:28
Now we're going to keep this small for faster training time so we can run more experiments. So we'll keep this small for faster training time. That's another little tidbit. Do you notice how quickly all of the cells ran above? Well, that's because we're using a relatively small data set. In the beginning, when you're running experiments, you want them to run quite quickly so that you can run them more often.
15:57:53
So you can learn more about your data so that you can try different things, try different models. So this is why we're using number of epochs equals three. We start with three so that our experiment runs in 30 seconds or a minute or so. That way, if something doesn't work, we haven't wasted so much time waiting for a model to train. Later on, we could train it for 100 epochs if we wanted to. So we're going to create a training and test loop.
15:58:18
So for epoch in TQDM range epochs, let's get this going. So for TQDM to work, we just wrap our iterator with TQDM and you'll see later on how this tracks the progress. So I'm going to put out a little print statement here. We'll go epoch. This is just going to say what epoch we're on. We'll go here. That's something that I like to do quite often is put little print statements here and there so that we know what's going on. So let's set up the training.
15:58:53
We're going to have to instantiate the train loss. We're going to set that to zero to begin with. And we're going to cumulatively add some values to the train loss here and then we'll see later on how this accumulates and we can calculate the training loss per batch. Let's what we're doing up here, calculate the train loss per batch. And then finally, at the end of the loop, we will divide our training loss by the number
15:59:17
of batches so we can get the average training loss per batch and that will give us the training loss per epoch. Now that's a lot of talking. If that doesn't make sense, remember. But if and out, code it out. So add a loop to loop through the training batches. So because our data is batchified now and I've got a crow or maybe a cooker bar sitting on the roof across from my apartment, it's singing its song this morning, lovely. So we're going to loop through our training batch data.
15:59:51
So I've got four batch, comma x, y, because remember our training batches come in the form of X. So that's our data or our images and why, which is label. You could call this image label or target as part of which would, but it's convention to often call your features X and your labels Y. We've seen this before in we're going to enumerate the train data loader as well. We do this so we can keep track of the number of batches we've been through. So that will give us batch there.
16:00:25
I'm going to set model zero to training mode because even though that's the default, we just want to make sure that it's in training mode. Now we're going to do the forward pass. If you remember, what are the steps in apply to our optimization loop? We do the forward pass. We calculate the loss of the minus zero grad, last backwards, up to minus a step, step, step. So let's do that. Hey, model zero, we'll put the features through there and then we're going to calculate the loss.
16:00:57
We've been through these steps before. So we're not going to spend too much time on the exact steps here, but we're just going to practice writing them out. And of course, later on, you might be thinking, then you'll, how come we haven't functionalized this training loop already? We've seemed to write the same generic code over and over again. Well, that's because we like to practice writing PyTorch code, right? We're going to functionalize them later on. Don't you worry about that.
16:01:20
So here's another little step that we haven't done before is we have the training loss. And so because we've set that to zero to begin with, we're going to accumulate the training loss values every batch. So we're going to just add it up here. And then later on, we're going to divide it by the total number of batches to get the average loss per batch. So you see how this loss calculation is within the batch loop here? So this means that one batch of data is going to go through the model.
16:01:49
And then we're going to calculate the loss on one batch of data. And this loop is going to continue until it's been through all of the batches in the train data loader. So 1875 steps or whatever there was. So accumulate train loss. And then we're going to optimize a zero grad, optimizer dot zero grad. And then number four is what? Loss backward. Loss backward. We'll do the back propagation step. And then finally, we've got number five, which is optimizer step.
16:02:27
So this is where I left my little note above to remind me and to also let you know, highlight that the optimizer will update a model's parameters once per batch rather than once per epoch. So you see how we've got a for loop inside our epoch loop here. So the batch loop. So this is what I meant that the optimizer, this is one of the advantages of using mini batches is not only is it more memory efficient because we're not loading 60,000 images into memory at a time.
16:02:57
We are updating our model's parameters once per batch rather than waiting for it to see the whole data set with every batch. Our model is hopefully getting slightly better. So that is because the optimizer dot step call is within the batch loop rather than the epoch loop. So let's now print out what's happening. Print out what's happening. So if batch, let's do it every 400 or so batches because we have a lot of batches.
16:03:32
We don't want to print out too often, otherwise we'll just fill our screen with numbers. That might not be a bad thing, but 400 seems a good number. That'll be about five printouts if we have 2000 batches. So print looked at, and of course you can adjust this to whatever you would like. That's the flexibility of PyTorch, flexibility of Python as well. So looked at how many samples have we looked at? So we're going to take the batch number, multiply it by X, the length of X is going
16:04:02
to be 32 because that is our batch size. Then we're going to just write down here the total number of items that we've got now of data set, and we can access that by going train data loader dot data set. So that's going to give us length of the data set contained within our train data loader, which is you might be able to guess 60,000 or should be. Now we have to, because we've been accumulating the train loss, this is going to be quite
16:04:37
high because we've been adding every single time we've calculated the loss, we've been adding it to the train loss, the overall value per batch. So now let's adjust if we wanted to find out, see how now we've got this line, we're outside of the batch loop. We want to adjust our training loss to get the average training loss per batch per epoch. So we're coming back to the epoch loop here. A little bit confusing, but you just line up where the loops are, and this is going to
16:05:06
help you figure out what context you're computing in. So now we are in the epoch loop. So divide total train loss by length of train data loader, oh, this is so exciting, training our biggest model yet. So train loss equals or divide equals, we're going to reassign the train loss, we're going to divide it by the length of the train data loader. So why do we do this? Well, because we've accumulated the train loss here for every batch in the train data
16:05:38
loader, but we want to average it out across how many batches there are in the train data loader. So this value will be quite high until we readjust it to find the average loss per epoch, because we are in the epoch loop. All right, there are a few steps going on, but that's all right, we'll figure this out, or what should happening in a minute, let's code up the testing loop. So testing, what do we have to do for testing? Well, let's set up a test loss variable.
16:06:06
Why don't we do accuracy for testing as well? Did we do accuracy for training? We didn't do accuracy for training, but that's all right, we'll stick to doing accuracy for testing. We'll go model zero dot eval, we'll put it in evaluation mode, and we'll turn on our inference mode context manager with torch dot inference mode. Now we'll do the same thing for x, y in test data loader, we don't need to keep track of the batches here again in the test data loader.
16:06:40
So we'll just loop through x, so features, images, and labels in our test data loader. We're going to do the forward pass, because the test loop, we don't have an optimization step, we are just passing our data through the model and evaluating the patterns it learned on the training data. So we're going to pass in x here. This might be a little bit confusing, let's do this x test, y test. That way we don't get confused with our x above for the training set.
16:07:12
Now we're going to calculate the loss, a cum, relatively might small that wrong app to sound that out. What do we have here? So we've got our test loss variable that we just assigned to zero above, just up here. So we're going to do test loss plus equals. We're doing this in one step here. Test spread, y test. So we're comparing our test prediction to our y test labels, our test labels. Now we're going to back out of the for loop here, because that's all we have to do, the
16:07:44
forward pass and calculate the loss for the test data set. Oh, I said we're going to calculate the accuracy. Silly me. So calculate accuracy. Let's go test act. And we've got plus equals. We can bring out our accuracy function here. That's what we downloaded from our helper functions dot pi before, y true equals y test. And then y pred equals test, pred dot arg max, dim equals one. Why do we do this? Well, because recall that the outputs of our model, the raw outputs of our model are going
16:08:18
to be logits and our accuracy function expects our true labels and our predictions to be in the same format. If our test pred is just logits, we have to call arg max to find the logit value with the highest index, and that will be the prediction label. And so then we're comparing labels to labels. That's what the arg max does here. So we can back out of the batch loop now, and we're going to now calculate Cal queue length, the test loss, average per batch.
16:08:58
So let's go here, test loss, divide equals length test data loader. So because we were in the context of the loop here of the batch loop, our test lost and test accuracy values are per batch and accumulated every single batch. So now we're just dividing them by how many batches we had, test data loader, and the same thing for the accuracy, calculate the ACK or test ACK average per batch. So this is giving us test loss and test accuracy per epoch, test ACK divided equals length,
16:09:39
test data loader, wonderful, we're so close to finishing this up. And now we'll come back to where's our epoch loop. We can, these lines are very helpful in Google CoLab, we scroll down. I believe if you want them, you can go settings or something like that, yeah, settings. That's where you can get these lines from if you don't have them. So print out what's happening. We are going to print f equals n, let's get the train loss in here. Ten loss and we'll print that to four decimal places.
16:10:16
And then we'll get the test loss, of course, test loss and we'll go, we'll get that to four decimal places as well. And then we'll get the test ACK, test accuracy, we'll get that to four decimal places as well. For f, wonderful. And then finally, one more step, ooh, we've written a lot of code in this video. We want to calculate the training time because that's another thing that we want to track. We want to see how long our model is taken to train.
16:10:51
So train time end on CPU is going to equal the timer and then we're going to get the total train time model zero so we can set up a variable for this so we can compare our modeling experiments later on. We're going to go print train time, start equals train time, start on CPU and equals train time end on CPU. And finally, the device is going to be string next model zero dot parameters. So we're just, this is one way of checking where our model zero parameters live. So beautiful, all right.
16:11:43
Have we got enough brackets there? I don't think we do. Okay. There we go. Whoo. I'll just show you what the output of this is. So next, model zero dot parameters, what does this give us? Oh, can we go device here? Oh, what do we have here? Model zero dot parameters. I thought this was a little trick. And then if we go next parameter containing. I thought we could get device, oh, there we go. Excuse me. That's how we get it. That's how we get the device that it's on. So let me just turn this.
16:12:41
This is what the output of that's going to be CPU. That's what we're after. So troubleshooting on the fly here. Hopefully all of this code works. So we went through all of our steps. We're looping through epochs at the top level here. We looped through the training batches, performed the training steps. So our training loop, forward pass, loss calculation, optimizer zero grad, loss backwards, calculate the loss per batch, accumulate those.
16:13:06
We do the same for the testing batches except without the optimizer steps and print out what's happening and we time it all for fun. A fair bit going on here, but if you don't think there's any errors, give that a go, run that code. I'm going to leave this one on a cliffhanger and we're going to see if this works in the next video. I'll see you there. Welcome back. The last video was pretty full on. We did a fair few steps, but this is all good practice.
16:13:35
The best way to learn PyTorch code is to write more PyTorch code. So did you try it out? Did you run this code? Did it work? Did we probably have an error somewhere? Well, let's find out together. You ready? Let's train our biggest model yet in three, two, one, bomb. Oh, of course we did. What do we have? What's going on? Indentation error. Ah classic. So print out what's happening. Do we not have an indent there? Oh, is that not in line with where it needs to be? Excuse me. Okay.
16:14:15
Why is this not in line? So this is strange to me, enter. How did this all get off by one? I'm not sure, but this is just what you'll face. Like sometimes you'll write this beautiful code that should work, but the main error of your entire code is that it's off by a single space. I'm not sure how that happened, but we're just going to pull this all into line. We could have done this by selecting it all, but we're going to do it line by line just
16:14:43
to make sure that everything's in the right order, beautiful, and we print out what's happening. Three, two, one, round two. We're going. Okay. So this is the progress bar I was talking about. Look at that. How beautiful is that? Oh, we're going quite quickly through all of our samples. I need to talk faster. Oh, there we go. We've got some good results. We've got the tests, the train loss, the test loss and the test accuracy is pretty darn good. Oh my goodness.
16:15:11
This is a good baseline already, 67%. So this is showing us it's about seven seconds per iteration. Remember TQDM is tracking how many epochs. We're going through. So we have three epochs and our print statement is just saying, hey, we've looked at zero out of 60,000 samples and we looked at 12,000 out of 60,000 samples and we finished on an epoch two because it's zero indexed and we have a train loss of 0.4550 and a test
16:15:41
loss 476 and a test accuracy 834265 and a training time about just over 21 seconds or just under 22. So keep in mind that your numbers may not be the exact same as mine. They should be in the same realm as mine, but due to inherent randomness of machine learning, even if we set the manual seed might be slightly different. So don't worry too much about that and what I mean by in the same realm, if your accuracy is 25 rather than 83, well then probably something's wrong there.
16:16:15
But if it's 83.6, well then that's not too bad. And the same with the train time on CPU, this will be heavily dependent, how long it takes to train will be heavily dependent on the hardware that you're using behind the scenes. So I'm using Google Colab Pro. Now that may mean I get a faster CPU than the free version of Google Colab. It also depends on what CPU is available in Google's computer warehouse where Google Colab is hosting of how fast this will be. So just keep that in mind.
16:16:49
If your time is 10 times that, then there's probably something wrong. If your time is 10 times less than that, well, hey, keep using that hardware because that's pretty darn good. So let's keep pushing forward. This will be our baseline that we try to improve upon. So we have an accuracy of 83.5 and we have a train time of 20 or so seconds. So we'll see what we can do with a model on the GPU later and then also later on a convolutional neural network.
16:17:18
So let's evaluate our model where we up to what we just did. We built a training loop. So we've done that. That was a fair bit of code. But now we're up to we fit the model to the data and make a prediction. Let's do these two combined, hey, we'll evaluate our model. So we'll come back. Number four is make predictions and get model zero results. Now we're going to create a function to do this because we want to build multiple models
16:17:47
and that way we can, if we have, say, model 0123, we can pass it to our function to evaluate that model and then we can compare the results later on. So that's something to keep in mind. If you're going to be writing a bunch of code multiple times, you probably want to functionize it and we could definitely do that for our training and last loops. But we'll see that later on. So let's go deaf of our model. So evaluate a given model, we'll pass it in a model, which will be a torch dot nn dot
16:18:19
module, what of type. And we'll pass it in a data loader, which will be of type torch dot utils dot data dot data loader. And then we'll pass in the loss function so that it can calculate the loss. We could pass in an evaluation metric if we wanted to track that too. So this will be torch nn dot module as well. And then, oh, there we go. Speaking of an evaluation function, let's pass in our accuracy function as well. And I don't want L, I want that.
16:18:54
So we want to return a dictionary containing the results of model predicting on data loader. So that's what we want. We're going to return a dictionary of model results. That way we could call this function multiple times with different models and different data loaders and then compare the dictionaries full of results depending on which model we passed in here. So let's set up loss and accuracy equals zero, zero, we'll start those off.
16:19:27
We'll go, this is going to be much the same as our testing loop above, except it's going to be functionalized and we're going to return a dictionary. So we'll turn on our context manager for inferencing with torch dot inference mode. Now we're going to loop through the data loader and we'll get the x and y values. So the x will be our data, the y will be our ideal labels, we'll make predictions with the model. In other words, do the forward pass. So we'll go y pred equals model on x.
16:19:57
Now we don't have to specify what model it is because we've got the model parameter up here. So we're starting to make our functions here or this function generalizable. So it could be used with almost any model and any data loader. So we want to accumulate the loss and accuracy values per batch because this is within the batch loop here per batch. And then we're going to go loss plus equals loss function, we'll pass it in the y pred
16:20:28
and the y the true label and we'll do the same with the accuracy. So except this time we use our accuracy function, we'll send in y true equals y and y pred equals y pred dot argmax because the raw outputs of our model are logits. And if we want to convert them into labels, we could take the softmax for the prediction probabilities, but we could also take the argmax and just by skipping the softmax step, the argmax will get the index where the highest value load it is, dim equals one.
16:21:03
And then we're going to make sure that we're still within the context manager here. So with torch inference mode, but outside the loop. So that'll be this line here. We're going to scale the loss and act to find the average loss slash act per batch. So loss will divide and assign to the length of the data loader. So that'll divide and reassign it to however many batches are in our data loader that we pass into our of our model function, then we'll do the same thing for the accuracy here.
16:21:41
Length data loader, beautiful. And now we're going to return a dictionary here. So return, we can return the model name by inspecting the model. We get an attribute of the model, which is its class name. I'll show you how you can do that. So this is helpful to track if you've created multiple different models and given them different class names, you can access the name attribute. So this only works when model was created with a class.
16:22:17
So you just have to ensure that your models have different class names. If you want to do it like that, because we're going to do it like that, we can set the model name to be its class name. We'll get the model loss, which is just this value here. After it's been scaled, we'll turn it into a single value by taking dot item. And then we'll go model dot act, or we'll get model underscore act for the models accuracy. We'll do the same thing here. Act.
16:22:42
I don't think we need to take the item because accuracy comes back in a different form. We'll find out, if in doubt, code it out. So calculate model zero results on test data set. And I want to let you know that you can create your own functions here to do almost whatever you want. I've just decided that this is going to be helpful for the models and the data that we're building. But keep that in mind that your models, your data sets might be different and will likely be different in the future.
16:23:09
So you can create these functions for whatever use case you need. Model zero results equals a vowel model. So we're just going to call our function that we've just created here. Model is going to equal model zero. The data loader is going to equal what? The test data loader, of course, because we want to evaluate it on the test data set. And we're going to send in our loss function, which is loss function that we assigned above just before our training loop.
16:23:42
If we come up here, our loss function is up here, and then if we go back down, we have our accuracy function is equal to our accuracy function. We just pass another function in there, beautiful. And let's see if this works. Model zero results. Did you see any typos likely or errors in our code? How do you think our model did? Well, let's find out. Oh, there we go. We got model accuracy. Can you see how we could reuse this dictionary later on?
16:24:15
So if we had model one results, model two results, we could use these dictionaries and compare them all together. So we've got our model name. Our version zero, the model has an accuracy of 83.42 and a loss of 0.47 on the test data loader. Again, your numbers may be slightly different. They should be in the same realm. But if they're not the exact same, don't worry too much. If they're 20 accuracy points less and the loss is 10 times higher, then you should probably
16:24:44
go back through your code and check if something is wrong. And I believe if we wanted to do a progress bar here, could we do that? TQDM. Let's have a look, eh? Oh, look at that progress bar. That's very nice. So that's nice and quick because it's only on 313 batches. It goes quite quick. So now, what's next? Well, we've built model one, we've got a model zero, sorry, I'm getting ahead myself. We've got a baseline here. We've got a way to evaluate our model. What's our workflow say?
16:25:16
So we've got our data ready. We've done that. We've picked or built a model. We've picked a loss function. We've built an optimizer. We've created a training loop. We've fit the model to the data. We've made a prediction. We've evaluated the model using loss and accuracy. We could evaluate it by making some predictions, but we'll save that for later on as in visualizing some predictions. I think we're up to improving through experimentation. So let's give that a go, hey?
16:25:41
Do you recall that we trained model zero on the CPU? How about we build model one and start to train it on the GPU? So in the next section, let's create number five, is set up device agnostic code. So we've done this one together for using a GPU if there is one. So my challenge to you for the next video is to set up some device agnostic code. So you might have to go into CoLab if you haven't got a GPU active, change runtime type
16:26:11
to GPU, and then because it might restart the runtime, you might have to rerun all of the cells above so that we get our helper functions file back and the data and whatnot. So set up some device agnostic code and I'll see you in the next video. How'd you go? You should give it a shot, did you set up some device agnostic code? I hope you gave it a go, but let's do it together. This won't take too long. The last two videos have been quite long.
16:26:38
So if I wanted to set device agnostic code, I want to see if I have a GPU available, do I? I can check it from the video SMI. That fails because I haven't activated a GPU in CoLab yet. I can also check here, torch CUDA is available. That will PyTorch will check if there's a GPU available with CUDA and it's not. So let's fix these two because we want to start using a GPU and we want to set up device agnostic code. So no matter what hardware our system is running, PyTorch leverages it.
16:27:12
So we're going to select GPU here, I'm going to click save and you'll notice that our Google CoLab notebook will start to reset and we'll start to connect. There we go. We've got a GPU on the back end, Python, three Google Compute Engine back end GPU. Do we have to reset this? NVIDIA SMI, wonderful, I have a Tesla T4 GPU with 16 gigabytes of memory, that is wonderful. And now do we have a GPU available? Oh, torch is not defined. Well, do you notice the numbers of these cells?
16:27:46
One, two, that means because we've reset our runtime to have a GPU, we have to rerun all the cells above. So we can go run before, that's going to run all the cells above, make sure that we download the data, make sure that we download the helper functions file, we go back up, we should see our data may be downloading. It shouldn't take too long. That is another advantage of using a relatively small data set that is already saved on PyTorch data sets.
16:28:14
Just keep in mind that if you use a larger data set and you have to re-download it into Google Colab, it may take a while to run, and if you build bigger models, they may take a while to run. So just keep that in mind for your experiments going forward, start small, increase when necessary. So we'll re-run this, we'll re-run this, and finally we're going to, oh, there we go, we've got a GPU, wonderful, but we'll write some device-agnostic code here, set up device-agnostic code.
16:28:42
So import-torch, now realistically you quite often do this at the start of every notebook, but I just wanted to highlight how we might do it if we're in the middle, and I wanted to practice running a model on a CPU only before stepping things up and going to a GPU. So device equals CUDA, this is for our device-agnostic code, if torch dot CUDA is available, and it looks like this is going to return true, else use the CPU. And then we're going to check device, wonderful, CUDA.
16:29:16
So we've got some device-agnostic code ready to go, I think it's time we built another model. And I asked the question before, do you think that the data set that we're working with requires nonlinearity? So the shirts, and the bags, and the shoes, do we need nonlinear functions to model this? Well it looks like our baseline model without nonlinearities did pretty well at modeling our data, so we've got a pretty good test accuracy value, so 83%, so out of 100 images
16:29:47
it predicts the right one, 83% of the time, 83 times out of 100, it did pretty well without nonlinearities. Why don't we try a model that uses nonlinearities and it runs on the GPU? So you might want to give that a go, see if you can create a model with nonlinear functions, try nn.relu, run it on the GPU, and see how it goes, otherwise we'll do it together in the next video, I'll see you there. Hello everyone, and welcome back, we are making some terrific progress, let's see how far
16:30:20
we've come, we've got a data set, we've prepared our data loaders, we've built a baseline model, and we've trained it, evaluated it, now it's time, oh, and the last video we set up device diagnostic code, but where are we in our little framework, we're up to improving through experimentation, and quite often that is building a different model and trying it out, it could be using more data, it could be tweaking a whole bunch of different things.
16:30:44
So let's get into some coding, I'm going to write it here, model one, I believe we're up to section six now, model one is going to be building a better model with nonlinearity, so I asked you to do the challenge in the last video to give it a go, to try and build a model with nonlinearity, I hope you gave it a go, because if anything that this course, I'm trying to impart on you in this course, it's to give things a go, to try things out
16:31:09
because that's what machine learning and coding is all about, trying things out, giving it a go, but let's write down here, we learned about the power of nonlinearity in notebook O2, so if we go to the learnpytorch.io book, we go to section number two, we'll just wait for this to load, and then if we come down here, we can search for nonlinearity, the missing piece nonlinearity, so I'm going to get this and just copy that in there, if you want to
16:31:41
see what nonlinearity helps us do, it helps us model nonlinear data, and in the case of a circle, can we model that with straight lines, in other words, linear lines? All linear means straight, nonlinear means non-straight, and so we learned that through the power of linear and nonlinear functions, neural networks can model almost any kind of data if we pair them in the right way, so you can go back through and read that there,
16:32:08
but I prefer to code things out and try it out on our data, so let's create a model with nonlinear and linear layers, but we also saw that our model with just linear layers can model our data, it's performing quite well, so that's where the experimentation side of things will come into play, sometimes you won't know what a model will do, whether it will work or won't work on your data set, but that is where we try different things
16:32:39
out, so we come up here, we look at our data, hmm, that looks actually quite linear to me as a bag, like it's just some straight lines, you could maybe model that with just straight lines, but there are some things which you could potentially classify as nonlinear in here, it's hard to tell without knowing, so let's give it a go, let's write a nonlinear model which is going to be quite similar to model zero here, except we're going to interspurse
16:33:06
some relu layers in between our linear layers, so recall that relu is a nonlinear activation function, and relu has the formula, if something comes in and it's a negative value, relu is going to turn that negative into a zero, and if something is positive, relu is just going to leave it there, so let's create another class here, fashion MNIST model V1, and we're going to subclass from nn.module, beautiful, and then we're going to initialize our model,
16:33:39
it's going to be quite the same as what we created before, we want an input shape, that's going to be an integer, and then we want a number of hidden units, and that's going to be an int here, and then we want an output shape, int, and I want to stress as well that although we're creating a class here with these inputs, classes are as flexible as functions, so if you need different use cases for your modeling classes, just keep that in mind that
16:34:08
you can build that functionality in, self dot layer stack, we're going to spell layer stack correctly, and we're going to set this equal to nn dot sequential, because we just want a sequential set of layers, the first one's going to be nn dot flatten, which is going to be flatten inputs into a single vector, and then we're going to go nn dot linear, because we want to flatten our stuff because we want it to be the right shape, if we don't
16:34:39
flatten it, we get shape issues, input shape, and then the out features of our linear layer is going to be the hidden units, hidden units, I'm just going to make some code cells here so that my code goes into the middle of the screen, then here is where we're going to add a nonlinear layer, so this is where we're going to add in a relu function, and where might we put these? Well, generally, you'll have a linear function followed by a nonlinear
16:35:08
function in the construction of neural networks. However, neural networks are as customizable as you can imagine, whether they work or not is a different question. So we'll go output shape here, as the out features, oh, do we miss this one up? Yes, we did. This needs to be hidden units. And why is that? Well, it's because the output shape of this linear layer here needs to match up with the input shape of this linear layer here. The relu
16:35:38
layer won't change the shape of our data. And you could test that out by printing the different shapes if you'd like. And then we're going to finish off with another nonlinear layer at the end. Relu. Now, do you think that this will improve our model's results or not? Well, it's hard to tell without trying it out, right? So let's continue building our model. We have to override the forward method. Self X is going to be, we'll give
16:36:05
a type in here, this is going to be a torch tensor as the input. And then we're just going to return what's happening here, we go self dot layer stack X. So that just means that X is going to pass through our layer stack here. And we could customize this, we could try it just with one nonlinear activation. This is actually our previous network, just with those commented out. All we've done is added in two relu functions. And so I'm
16:36:31
going to run that beautiful. And so what should we do next? Well, we shouldn't stand shaded but previously we ran our last model model zero on if we go parameters. Do we run this on the GPU or the CPU? On the CPU. So how about we try out our fashion MNIST model or V one running on the device that we just set up which should be CUDA. Wonderful. So we can instantiate. So create an instance of model one. So we want model one or actually
16:37:09
we'll set up a manual seed here so that whenever we create a new instance of a model, it's going to be instantiated with random numbers. We don't necessarily have to set a random seed, but we do so anyway so that our values are quite similar on your end and my end input shape is going to be 784. Where does that come from? Well, that's because this is the output of the flatten layer after our 28 by 28 image goes in. Then we're going to set
16:37:42
up the hidden units. We're going to use the same number of hidden units as before, which is going to be 10. And then the output shape is what? We need one value, one output neuron for each of our classes. So length of the class names. And then we're going to send this to the target device so we can write send to the GPU if it's available. So now that we've set up device agnostic code in the last video, we can just put two device
16:38:08
instead of hard coding that. And so if we check, so this was the output for model zero's device, let's now check model one's device, model one parameters, and we can check where those parameters live by using the device attribute. Beautiful. So our model one is now living on the GPU CUDA at index zero. Index zero means that it's on the first GPU that we have available. We only have one GPU available. So it's on this Tesla T for GPU. Now, we've
16:38:44
got a couple more things to do. Now that we've created another model, we can recreate if we go back to our workflow, we've just built a model here. What do we have to do after we built a model? We have to instantiate a loss function and an optimizer. Now we've done both of those things for model zero. So that's what we're going to do in the next video. But I'd like you to go ahead and try to create a loss function for our model and
16:39:07
optimizer for model one. The hint is that they can be the exact same loss function and optimizer as model zero. So give that a shot and I'll see you in the next video. Welcome back. In the last video, we created another model. So we're continuing with our modeling experiments. And the only difference here between fashion MNIST model V1 and V0 is that we've added in nonlinear layers. Now we don't know for now we could think or guess whether
16:39:35
they would help improve our model. And with practice, you can start to understand how different functions will influence your neural networks. But I prefer to, if in doubt, code it out, run lots of different experiments. So let's continue. We now have to create a loss function, loss, optimizer, and evaluation metrics. So we've done this for model zero. So we're not going to spend too much time explaining what's going on here. And we've
16:40:01
done this a fair few times now. So from helper functions, which is the script we downloaded before, we're going to import our accuracy function. And we're going to set up a loss function, which is we're working with multi class classification. So what loss function do we typically use? And then dot cross entropy loss. And as our optimizer is going to be torch dot opt in dot SGD. And we're going to optimize this time. I'll put in the params
16:40:31
keyword here, model one dot parameters. And the learning rate, we're just going to keep it the same as our previous model. And that's a thing to keep a note for your experiments. When you're running fair few experiments, you only really want to tweak a couple of things or maybe just one thing per experiment, that way you can really narrow down what actually influences your model and what improves it slash what doesn't improve it. And a little
16:40:55
pop quiz. What does a loss function do? This is going to measure how wrong our model is. And what does the optimizer do? Tries to update our models parameters to reduce the loss. So that's what these two functions are going to be doing. The accuracy function is of a course going to be measuring our models accuracy. We measure the accuracy because that's one of the base classification metrics. So we'll run this. Now what's next? We're getting
16:41:33
quite good at this. We've picked a loss function and an optimizer. Now we're going to build a training loop. However, we spent quite a bit of time doing that in a previous video. If we go up here, that was our vowel model function. Oh, that was helpful. We turned it into a function. How about we do the same with these? Why don't we make a function for our training loop as well as our testing loop? So I think you can give this a go. We're going
16:42:03
to make a function in the next video for training. We're going to call that train step. And we'll create a function for testing called test step. Now they'll both have to take in some parameters. I'll let you figure out what they are. But otherwise, we're going to code that up together in the next video. So I'll see you there. So we've got a loss function ready and an optimizer. What's our next step? Well, it's to create training and evaluation loops. So let's make a heading here. We're going to
16:42:32
call this functionizing training and evaluation or slash testing loops because we've written similar code quite often for training and evaluating slash testing our models. Now we're going to start moving towards functionizing code that we've written before because that's not only a best practice, it helps reduce errors because if you're writing a training loop all the time, we may get it wrong. If we've got one that works for our particular
16:43:01
problem, hey, we might as well save that as a function so we can continually call that over and over and over again. So how about we, and this is going to be very rare that I'm going to allow you to do this is that is we're going to copy this training and you might have already attempted to create this. That is the function called, let's create a function for one training loop. And we're going to call this train step. And we're going
16:43:34
to create a function for the testing loop. You're going to call this test step. Now these are just what I'm calling them. You can call them whatever you want. I just understand it quite easily by calling it train step. And then we can for each epoch in a range, we call our training step. And then the same thing for each epoch in a range, we can call a testing step. This will make a lot more sense once we've coded it out. So let's put
16:44:01
the training code here. To functionize this, let's start it off with train step. Now what parameters should our train step function take in? Well, let's think about this. We need a model. We need a data loader. We need a loss function. And we need an optimizer. We could also put in an accuracy function here if we wanted to. And potentially it's not here, but we could put in what target device we'd like to compute on and make our
16:44:33
code device agnostic. So this is just the exact same code we went through before. We loop through a data loader. We do the forward pass. We calculate the loss. We accumulate it. We zero the optimizer. We perform backpropagation in respect to the loss with the parameters of the model. And then we step the optimizer to hopefully improve the parameters of our model to better predict the data that we're trying to predict. So let's craft a train
16:45:00
step function here. We'll take a model, which is going to be torch nn.module, type hint. And we're going to put in a data loader, which is going to be of type torch utils dot data dot data loader. Now we don't necessarily need to put this in these type hints, but they're relatively new addition to Python. And so you might start to see them more and more. And it also just helps people understand what your code is expecting. So the loss
16:45:30
function, we're going to put in an optimizer torch dot opt in, which is a type optimizer. We also want an accuracy function. We don't necessarily need this either. These are a lot of nice to habs. The first four are probably the most important. And then the device. So torch is going to be torch dot device equals device. So we'll just hard code that to be our already set device parameter. And we'll just write in here, performs training step
16:46:04
with model, trying to learn on data loader. Nice and simple, we could make that more explanatory if we wanted to, but we'll leave it at that for now. And so right at the start, we're going to set up train loss and train act equals zero zero. We're going to introduce accuracy here. So we can get rid of this. Let's just go through this line by line. What do we need to do here? Well, we've got four batch XY in enumerate train data loader. But
16:46:37
we're going to change that to data loader up here. So we can just change this to data loader. Wonderful. And now we've got model zero dot train. Do we want that? Well, no, because we're going to keep this model agnostic, we want to be able to use any model with this function. So let's get rid of this model dot train. We are missing one step here is put data on target device. And we could actually put this model dot train up here. Put model
16:47:10
into training mode. Now, this will be the default for the model. But just in case we're going to call it anyway, model dot train, put data on the target device. So we're going to go XY equals X dot two device, Y dot two device. Wonderful. And the forward pass, we don't need to use model zero anymore. We're just going to use model that's up here. The loss function can stay the same because we're passing in a loss function up there. The train
16:47:42
loss can be accumulated. That's fine. But we might also accumulate now the train accuracy, limit loss, and accuracy per batch. So train act equals or plus equals our accuracy function on Y true equals Y and Y pred equals Y pred. So the outputs here, Y pred, we need to take because the raw outputs, outputs, the raw logits from the model, because our accuracy function expects our predictions to be in the same format as our true values. We need
16:48:20
to make sure that they are we can call the argmax here on the first dimension. This is going to go from logits to prediction labels. We can keep the optimizer zero grab the same because we're passing in an optimizer up here. We can keep the loss backwards because the loss is just calculated there. We can keep optimizer step. And we could print out what's happening. But we might change this up a little bit. We need to divide the total train loss
16:48:50
and accuracy. I just want to type in accuracy here because now we've added in accuracy metric act. So train act divided equals length train data loader. Oh, no, sorry. We can just use the data loader here, data loader, data loader. And we're not going to print out per batch here. I'm just going to get rid of this. We'll make at the end of this step, we will make our print out here, print. Notice how it's at the end of the step because we're outside
16:49:23
the for loop now. So we're going to here, we're accumulating the loss on the training data set and the accuracy on the training data set per batch. And then we're finding out at the end of the training steps. So after it's been through all the batches in the data loader, we're finding out what the average loss is per batch. And the average accuracy is per batch. And now we're going to go train loss is going to be the train
16:49:53
loss on 0.5. And then we're going to go train act is going to be train act. And we're going to set that to 0.2 F. Get that there, percentage. Wonderful. So if all this works, we should be able to call our train step function and pass it in a model, a data loader, a loss function, an optimizer, an accuracy function and a device. And it should automatically do all of these steps. So we're going to find that out in a later video. In the next video,
16:50:34
we're going to do the same thing we've just done for the training loop with the test step. But here's your challenge for this video is to go up to the testing loop code we wrote before and try to recreate the test step function in the same format that we've done here. So give that a go. And I'll see you in the next video. Welcome back. In the last video, we functionalized our training loop. So now we can call this train step function. And instead
16:51:01
of writing all this training loop code again, well, we can train our model through the art of a function. Now let's do the same for our testing loop. So I issued you the challenge in the last video to give it a go. I hope you did because that's the best way to practice PyTorch code is to write more pytorch code. Let's put in a model, which is going to be torch and then dot module. And we're going to put in a data loader. Because we need a
16:51:28
model and we need data, the data loader is going to be, of course, the test data load here, torch dot utils dot data dot data loader. And then we're going to put in a loss function, which is going to be torch and end up module as well. Because we're going to use an end up cross entropy loss. We'll see that later on. We're going to put in an accuracy function. We don't need an optimizer because we're not doing any optimization in the testing loop.
16:51:53
We're just evaluating. And the device can be torch dot device. And we're going to set that as a default to the target device parameter. Beautiful. So we'll put a little doctoring here. So performs a testing loop step on model going over data loader. Wonderful. So now let's set up a test loss and a test accuracy, because we'll measure test loss and accuracy without testing loop function. And we're going to set the model into, I'll just put a comment
16:52:28
here, put the model in a vowel mode. So model dot a vowel, we don't have to use any underscore here as in model zero, because we have a model coming in the top here. Now, what should we do? Well, because we're performing a test step, we should turn on inference mode. So turn on inference mode, inference mode context manager. Remember, whenever you're performing predictions with your model, you should put it in model dot a vowel. And if you want as
16:53:02
many speedups as you can get, make sure the predictions are done within the inference mode. Because remember, inference is another word for predictions within the inference mode context manager. So we're going to loop through our data loader for X and Y in data loader. We don't have to specify that this is X test. For Y test, we could if we wanted to. But because we're in another function here, we can just go for X, Y in data loader,
16:53:31
we can do the forward pass. After we send the data to the target device, target device, so we're going to have X, Y equals X dot two device. And the same thing with Y, we're just doing best practice here, creating device agnostic code. Then what should we do? Well, we should do the thing that I said before, which is the forward pass. Now that our data and model be on the same device, we can create a variable here test pred equals model, we're
16:54:02
going to pass in X. And then what do we do? We can calculate the loss. So to calculate the loss slash accuracy, we're going to accumulate it per batch. So we'll set up test loss equals loss function. Oh, plus equals loss function. We're going to pass it in test pred and Y, which is our truth label. And then the test act where you will accumulate as well, using our accuracy function, we'll pass in Y true equals Y. And then Y pred, what do we have
16:54:36
to do to Y pred? Well, our test pred, we have to take the argmax to convert it from. So this is going to outputs raw logits. Remember, a models raw output is referred to as logits. And then here, we have to go from logits to prediction labels. Beautiful. Oh, little typo here. Did you catch that one? Tab, tab. Beautiful. Oh, look how good this function is looking. Now we're going to adjust the metrics. So adjust metrics and print out. You might notice
16:55:14
that we're outside of the batch loop here, right? So if we draw down from this line for and we write some code here, we're still within the context manager. This is important because if we want to adapt a value created inside the context manager, we have to modify it still with inside that context manager, otherwise pytorch will throw an error. So try to write this code if you want outside the context manager and see if it still works. So test loss, we're
16:55:46
going to adjust it to find out the average test loss and test accuracy per batch across a whole step. So we're going to go length data loader. Now we're going to print out what's happening. Print out what's happening. So test loss, which we put in here, well, we're going to get the test loss. Let's get this to five decimal places. And then we're going to go test act. And we will get that to two decimal places. You could do this as
16:56:18
many decimal as you want. You could even times it by 100 to get it in proper accuracy format. And we'll put a new line on the end here. Wonderful. So now it looks like we've got functions. I haven't run this cell yet for a training step and a test step. So how do you think we could replicate if we go back up to our training loop that we wrote before? How do you think we could replicate the functionality of this, except this time using our functions? Well,
16:56:51
we could still use this for epoch and TQDM range epochs. But then we would just call our training step for this training code, our training step function. And we would call our testing step function, passing in the appropriate parameters for our testing loop. So that's what we'll do in the next video. We will leverage our two functions, train step and test step to train model one. But here's your challenge for this video. Give
16:57:18
that a go. So use our training step and test step function to train model one for three epochs and see how you go. But we'll do it together in the next video. Welcome back. How'd you go? Did you create a training loop or a PyTorch optimization loop using our training step function and a test step function? Were there any errors? In fact, I don't even know. But how about we find out together? Hey, how do we combine these two functions to create
16:57:46
an optimization loop? So I'm going to go torch dot manual seed 42. And I'm going to measure the time of how long our training and test loop takes. This time we're using a different model. So this model uses nonlinearities and it's on the GPU. So that's the main thing we want to compare is how long our model took on CPU versus GPU. So I'm going to import from time it, import default timer as timer. And I'm going to start the train time. Train
16:58:16
time start on GPU equals timer. And then I'm just right here, set epochs. I'm going to set epochs equal to three, because we want to keep our training experiments as close to the same as possible. So we can see what little changes do what. And then it's create a optimization and evaluation loop using train step and test step. So we're going to loop through the epochs for epoch in TQDM. So we get a nice progress bar in epochs. Then we're
16:58:59
going to print epoch. A little print out of what's going on. Epoch. And we'll get a new line. And then maybe one, two, three, four, five, six, seven, eight or something like that. Maybe I'm miscounted there. But that's all right. Train step. What do we have to do for this? Now we have a little doc string. We have a model. What model would we like to use? We'd like to use model one. We have a data loader. What data loader would we
16:59:26
like to use? Well, we'd like to use our train data loader. We also have a loss function, which is our loss function. We have an optimizer, which is our optimizer. And we have an accuracy function, which is our accuracy function. And oops, forgot to put FM. And finally, we have a device, which equals device, but we're going to set that anyway. So how beautiful is that for creating a training loop? Thanks to the code that we've functionalized before. And
17:00:02
just recall, we set our optimizer and loss function in a previous video. You could bring these down here if you really wanted to, so that they're all in one place, either way up. But we can just get rid of that because we've already set it. Now we're going to do the same thing for our test step. So what do we need here? Let's check the doc string. We could put a little bit more information in this doc string if we wanted to to really
17:00:25
make our code more reusable, and so that if someone else was to use our code, or even us in the future knows what's going on. But let's just code it out because we're just still fresh in our minds. Model equals model one. What's our data loader going to be for the test step? It's going to be our test data loader. Then we're going to set in a loss function, which is going to be just the same loss function. We don't need to use an optimizer
17:00:49
here because we are only evaluating our model, but we can pass in our accuracy function. Accuracy function. And then finally, the device is already set, but we can just pass it in anyway. Look at that. Our whole optimization loop in a few lines of code. Isn't that beautiful? So these functions are something that you could put in, like our helper functions dot pi. And that way you could just import it later on. And you don't have to write your
17:01:17
training loops all over again. But we'll see a more of an example of that later on in the course. So let's keep going. We want to measure the train time, right? So we're going to create, once it's been through these steps, we're going to create train time end on CPU. And then we're going to set that to the timer. So all this is going to do is measure at value in time, once this line of code is run, it's going to run all of these
17:01:46
lines of code. So it's going to perform the training and optimization loop. And then it's going to, oh, excuse me, this should be GPU. It's going to measure a point in time here. So once all this codes run, measure a point in time there. And then finally, we can go total train time for model one is equal to print train time, which is our function that we wrote before. And we pass it in a start time. And it prints the difference between
17:02:14
the start and end time on a target device. So let's do that. Start equals what? Train time start on GPU. The end is going to be train time end on GPU. And the device is going to be device. Beautiful. So are you ready to run our next modeling experiment model one? We've got a model running on the GPU, and it's using nonlinear layers. And we want to compare it to our first model, which our results were model zero results. And we have total
17:02:53
train time on model zero. Yes, we do. So this is what we're going for. Does our model one beat these results? And does it beat this result here? So three, two, one, do we have any errors? No, we don't. Okay. Train step got an unexpected keyword loss. Oh, did you catch that? I didn't type in loss function. Let's run it again. There we go. Okay, we're running. We've got a progress bar. It's going to output at the end of each epoch. There
17:03:25
we go. Training loss. All right. Test accuracy, training accuracy. This is so exciting. I love watching neural networks train. Okay, we're improving per epoch. That's a good sign. But we've still got a fair way to go. Oh, okay. So what do we have here? Well, we didn't beat our, hmm, it looks like we didn't beat our model zero results with the nonlinear layers. And we only just slightly had a faster training time. Now, again, your numbers might
17:03:58
not be the exact same as what I've got here. Right? So that's a big thing about machine learning is that it uses randomness. So your numbers might be slightly different. The direction should be quite similar. And we may be using different GPUs. So just keep that in mind. Right now I'm using a new video, SMI. I'm using a Tesla T4, which is at the time of recording this video, Wednesday, April 20, 2022 is a relatively fast GPU for making
17:04:25
inference. So just keep that in mind. Your GPU in the future may be different. And your CPU that you run may also have a different time here. So if these numbers are like 10 times higher, you might want to look into seeing if your code is there's some error. If they're 10 times lower, well, hey, you're running it on some fast hardware. So it looks like my code is running on CUDA slightly faster than the CPU, but not dramatically faster.
17:04:52
And that's probably akin to the fact that our data set isn't too complex and our model isn't too large. What I mean by that is our model doesn't have like a vast amount of layers. And our data set is only comprised of like, this is the layers our model has. And our data set is only comprised of 60,000 images that are 28 by 28. So as you can imagine, the more parameters in your model, the more features in your data, the higher this time
17:05:18
is going to be. And you might sometimes even find that your model is faster on CPU. So this is the train time on CPU. You might sometimes find that your model's training time on a CPU is in fact faster for the exact same code running on a GPU. Now, why might that be? Well, let's write down this here. Let's go note. Sometimes, depending on your data slash hardware, you might find that your model trains faster on CPU than GPU. Now,
17:06:00
why is this? So one of the number one reasons is that one, it could be that the overhead for copying data slash model to and from the GPU outweighs the compute benefits offered by the GPU. So that's probably one of the number one reasons is that you have to, for data to be processed on a GPU, you have to copy it because it is by default on the CPU. If you have to copy it to that GPU, you have some overhead time for doing that copy into
17:06:40
the GPU memory. And then although the GPU will probably compute faster on that data once it's there, you still have that back and forth of going between the CPU and the GPU. And the number two reason is that the hardware you're using has a better CPU in terms of compute capability than the GPU. Now, this is quite a bit rarer. Usually if you're using a GPU like a fairly modern GPU, it will be faster at computing, deep learning
17:07:14
or running deep learning algorithms than your general CPU. But sometimes these numbers of compute time are really dependent on the hardware that you're running. So you'll get the biggest benefits of speedups on the GPU when you're running larger models, larger data sets, and more compute intensive layers in your neural networks. And so if you'd like a great article on how to get the most out of your GPUs, it's a little bit technical,
17:07:39
but this is something to keep in mind as you progress as a machine learning engineer is how to make your GPUs go burr. And I mean that burr from first principles. There we go. Making deep learning go burr as in your GPU is going burr because it's running so fast from first principles. So this is by Horace He who works on PyTorch. And it's great. It talks about compute as a first principle. So here's what I mean by copying
17:08:13
memory and compute. There might be a fair few things you're not familiar with here, but that's okay. But just be aware bandwidth. So bandwidth costs are essentially the cost paid to move data from one place to another. That's what I was talking about copying stuff from the CPU to the GPU. And then also there's one more, where is it overhead? Overhead is basically everything else. I called it overhead. There are different terms for different things.
17:08:37
This article is excellent. So I'm going to just copy this in here. And you'll find this in the resources, by the way. So for more on how to make your models compute faster, see here. Lovely. So right now our baseline model is performing the best in terms of results. And in terms of, or actually our model computing on the GPU is performing faster than our CPU. Again yours might be slightly different. For my case, for my particular hardware, CUDA
17:09:10
is faster. Except model zero, our baseline is better than model one. So what's to do next? Well, it's to keep experimenting, of course. I'll see you in the next video. Welcome back. Now, before we move on to the next modeling experiment, let's get a results dictionary for our model one, a model that we trained on. So just like we've got one for model zero, let's create one of these for model one results. And we can create that without a vowel model
17:09:39
function. So we'll go right back down to where we were. I'll just get rid of this cell. And let's type in here, get model one results dictionary. This is helpful. So later on, we can compare all of our modeling results, because they'll all be in dictionary format. So we're going to model one results equals a vowel model on a model equals model one. And we can pass in a data loader, which is going to be our test data loader. Then we
17:10:12
can pass in a loss function, which is going to equal our loss function. And we can pass in our accuracy function equals accuracy function. Wonderful. And then if we check out our model one results, what do we get? Oh, no, we get an error. Do we get the code right? That looks right to me. Oh, what does this say runtime error expected all tensors to be on the same device, but found at least two devices, CUDA and CPU. Of course. So why did this happen?
17:10:49
Well, let's go back up to our of our model function, wherever we defined that. Here we go. Ah, I see. So this is a little gotcha in pytorch or in deep learning in general. There's a saying in the industry that deep learning models fail silently. And this is kind of one of those ones. It's because our data and our model are on different devices. So remember how I said the three big errors are shape mismatches with your data and your model device
17:11:19
mismatches, which is what we've got so far. And then data type mismatches, which is if your data is in the wrong data type to be computed on. So what we're going to have to do to fix this is let's bring down our vowel model function down to where we were. And just like we've done in our test step and train step functions, where we've created device agnostic data here, we've sent our data to the target device, we'll do that exact
17:11:47
same thing in our vowel model function. And this is just a note for going forward. It's always handy to where you can create device agnostic code. So we've got our new of our model function here for x, y in our data loader. Let's make our data device agnostic. So just like our model is device agnostic, we've sent it to the target device, we will do the same here, x dot two device, and then y dot two device. Let's see if that works. We will
17:12:20
just rerun this cell up here. I'll grab this, we're just going to write the exact same code as what we did before. But now it should work because we've sent our, we could actually also just pass in the target device here, device equals device. That way we can pass in whatever device we want to run it on. And we're going to just add in device here, device equals device. And let's see if this runs correctly. Beautiful. So if we compare
17:12:50
this to our model zero results, it looks like our baseline's still out in front. But that's okay. We're going to in the next video, start to step things up a notch and move on to convolutional neural networks. This is very exciting. And by the way, just remember, if your numbers here aren't exactly the same as mine, don't worry too much. If they're out landishly different, just go back through your code and see if it's maybe a cell hasn't been run correctly
17:13:16
or something like that. If there are a few decimal places off, that's okay. That's due to the inherent randomness of machine learning and deep learning. But with that being said, I'll see you in the next video. Let's get our hands on convolutional neural networks. Welcome back. In the last video, we saw that our second modeling experiment, model one, didn't quite beat our baseline. But now we're going to keep going with modeling experiments.
17:13:42
And we're going to move on to model two. And this is very exciting. We're going to build a convolutional neural network, which are also known as CNN. CNNs are also known as com net. And CNNs are known for their capabilities to find patterns in visual data. So what are we going to do? Well, let's jump back into the keynote. We had a look at this slide before where this is the typical architecture of a CNN. There's a fair bit going on here, but
17:14:18
we're going to step through it one by one. We have an input layer, just like any other deep learning model. We have to input some kind of data. We have a bunch of hidden layers in our case in a convolutional neural network, you have convolutional layers. You often have hidden activations or nonlinear activation layers. You might have a pooling layer. You generally always have an output layer of some sort, which is usually a linear layer. And
17:14:44
so the values for each of these different layers will depend on the problem you're working on. So we're going to work towards building something like this. And you'll notice that a lot of the code is quite similar to the code that we've been writing before for other PyTorch models. The only difference is in here is that we're going to use different layer types. And so if we want to visualize a CNN in a colored block edition, we're going
17:15:09
to code this out in a minute. So don't worry too much. We have a simple CNN. You might have an input, which could be this image of my dad eating some pizza with two thumbs up. We're going to preprocess that input. We're going to, in other words, turn it into a tensor in red, green and blue for an image. And then we're going to pass it through a combination of convolutional layers, relu layers and pooling layers. Now again, this
17:15:36
is a thing to note about deep learning models. I don't want you to get too bogged down in the order of how these layers go, because they can be combined in many different ways. In fact, research is coming out almost every day, every week about how to best construct these layers. The overall principle is what's more important is how do you get your inputs into an idolized output? That's the fun part. And then of course, we have the linear output
17:16:01
layer, which is going to output however many classes or value for however many classes that we have in the case of classification. And then if you want to make your CNN deeper, this is where the deep comes from deep learning, you can add more layers. So the theory behind this, or the practice behind this, is that the more layers you add to your deep learning model, the more chances it has to find patterns in the data. Now, how does it find these patterns?
17:16:30
Well, each one of these layers here is going to perform, just like what we've seen before, a different combination of mathematical operations on whatever data we feed it. And each subsequent layer receives its input from the previous layer. In this case, there are some advanced networks that you'll probably come across later in your research and machine learning career that use inputs from layers that are kind of over here or the way down here or
17:16:57
something like that. They're known as residual connections. But that's beyond the scope of what we're covering for now. We just want to build our first convolutional neural network. And so let's go back to Google Chrome. I'm going to show you my favorite website to learn about convolutional neural networks. It is the CNN explainer website. And this is going to be part of your extra curriculum for this video is to spend 20 minutes clicking and
17:17:22
going through this entire website. We're not going to do that together because I would like you to explore it yourself. That is the best way to learn. So what you'll notice up here is we have some images of some different sort. And this is going to be our input. So let's start with pizza. And then we have a convolutional layer, a relu layer, a conv layer, a relu layer, max pool layer, com to relu to com to relu to max pool to this
17:17:47
architecture is a convolutional neural network. And it's running live in the browser. And so we pass this image, you'll notice that it breaks down into red, green and blue. And then it goes through each of these layers and something happens. And then finally, we have an output. And you notice that the output has 10 different classes here, because we have one, two, three, four, five, six, seven, eight, nine, 10, different classes of image
17:18:14
in this demo here. And of course, we could change this if we had 100 classes, we might change this to 100. But the pieces of the puzzle here would still stay quite the same. And you'll notice that the class pizza has the highest output value here, because our images of pizza, if we change to what is this one, espresso, it's got the highest value there. So this is a pretty well performing convolutional neural network. Then we have
17:18:40
a sport car. Now, if we clicked on each one of these, something is going to happen. Let's find out. We have a convolutional layer. So we have an input of an image here that 64 64 by three. This is color channels last format. So we have a kernel. And this kernel, this is what happens inside a convolutional layer. And you might be going, well, there's a lot going on here. And yes, of course, there is if this is the first time you ever seen this.
17:19:06
But essentially, what's happening is a kernel, which is also known as a filter, is going over our image pixel values, because of course, they will be in the format of a tensor. And trying to find small little intricate patterns in that data. So if we have a look here, and this is why it's so valuable to go through this and just play around with it, we start in a top left corner, and then slowly move along, you'll see on the output on the right
17:19:29
hand side, we have another little square. And do you notice in the middle all of those numbers changing? Well, that is the mathematical operation that's happening as a convolutional layer convolves over our input image. How cool is that? And you might be able to see on the output there that there's some slight values for like, look around the headlight here. Do you notice on the right how there's some activation? There's some red tiles there? Well, that
17:19:57
just means that potentially this layer or this hidden unit, and I want to zoom out for a second, is we have 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 hidden units. Each one of these is going to learn a different feature about the data. And now the beauty of deep learning, but also one of the curses of deep learning is that we don't actually control what each one of these learns. The magic of deep learning is that it figures it out itself what is
17:20:26
best to learn. We go into here, notice that each one we click on has a different representation on the right hand side. And so this is what's going to happen layer by layer as it goes through the convolutional neural network. And so if you want to read about what is a convolutional neural network, you can go through here. But we're going to replicate this exact neural network here with PyTorch code. That's how I'd prefer to learn it. But if you want the
17:20:51
intuition behind it, the math behind it, you can check out all of these resources here. That is your extra curriculum for this video. So we have an input layer, we have a convolutional layer, you can see how the input gets modified by some sort of mathematical operation, which is of course, the convolutional operation. And we have there all different numbers finding different patterns and data. This is a really good example here. You notice that the outputs
17:21:17
eyes slightly changes, that'll be a trend throughout each layer. And then we can understand the different hyper parameters, but I'm going to leave this for you to explore on your own. In the next video, we're going to start to write PyTorch code to replicate everything that's going on here. So I'm going to link this in here to find out what's happening inside CNN. See this website here. So join me in the next video. This is super exciting.
17:21:50
We're going to build our first convolutional neural network for computer vision. I'll see you there. Welcome back. In the last video, we went briefly through the CNN explainer website, which is my favorite resource for learning about convolutional neural networks. And of course, we could spend 20 minutes clicking through everything here to find out what's going on with a convolutional neural network, or we could start to code one up. So how about
17:22:17
we do that? Hey, if and down, code it out. So we're going to create a convolutional neural network. And what I'm going to do is I'm going to build this, or we're going to build this model together in this video. And then because it's going to use layers or PyTorch layers that we haven't looked at before, we're going to spend the next couple of videos stepping through those layers. So just bear with me, as we code this entire model together, we'll
17:22:45
go break it down in subsequent videos. So let's build our first convolutional neural network. That's a mouthful, by the way, I'm just going to probably stick to saying CNN. Fashion MNIST, we're up to model V2. We're going to subclass nn.module, as we always do when we're building a PyTorch model. And in here, we're going to say model architecture
17:23:10
that replicates the tiny VGG. And you might be thinking, where did you get that from, Daniel? Model from CNN explainer website. And so oftentimes, when convolutional neural networks or new types of architecture come out, the authors of the research paper that present the model get to name the model. And so that way, in the future, you can refer to different types of model architectures with just a simple name, like tiny VGG. And people kind of know what's going on.
17:23:42
So I believe somewhere on here, it's called tiny VGG, tiny VGG. We have nothing. Yeah, there we go. In tiny VGG. And do we have more than one tiny, tiny, yeah, tiny VGG. And if we look up VGG, conv net, VGG 16 was one of the original ones, VGG, very deep convolutional neural networks of VGG net. There's also ResNet, which is another convolutional neural network.
17:24:16
You can also, I don't want to give you my location, Google, you can go popular CNN architectures. And this will give you a fair few options. Lynette is one of the first AlexNet, ZF net, whole bunch of different resources. And also, how could you find out more about a convolutional neural network? What is a convolutional neural network? You can go through that. But let's stop that for a moment. Let's code this one up together. So we're going to initialize our
17:24:44
class here, def init. We're going to pass it in an input shape, just like we often do. We're going to put in a number of hidden units, which is an int. And we're going to put in an output shape, which is an int. Wonderful. So nothing to outlandish that we haven't seen before there. And we're going to go super dot init to initialize our initializer for lack of a better way of putting it. Now, we're going to create our neural network in a couple of blocks this time. And
17:25:18
you might often hear in when you learn more about convolutional neural networks, or I'll just tell you that things are referred to are often referred to as convolutional blocks. So if we go back to our keynote, this here, this combination of layers might be referred to as a convolutional block. And a convolutional block, a deeper CNN, might be comprised of multiple convolutional blocks. So to add to the confusion, a block is comprised of multiple layers. And then an overall architecture
17:25:50
is comprised of multiple blocks. And so the deeper and deeper your models get, the more blocks it might be comprised of, and the more layers those blocks may be comprised of within them. So it's kind of like Lego, which is very fun. So let's put together an an ensequential.
17:26:09
Now, the first few layers here that we're going to create in conv block one, uh, nn.com 2d. Oh, look at that. Us writing us our first CNN layer. And we have to define something here, which is in channels. So this channels refers to the number of channels in your visual data. And we're going to put in input shape. So we're defining the input shape. This is going to be the first layer in our model. The input shape is going to be what we define when we instantiate
17:26:39
this class. And then the out channels. Oh, what's the out channels going to be? Well, it's going to be hidden units, just like we've done with our previous models. Now the difference here is that in nn.com 2d, we have a number of different hyper parameters that we can set. I'm going to set some pretty quickly here, but then we're going to step back through them, not only in this video, but in subsequent videos. We've got a fair bit going on here.
17:27:04
We've got in channels, which is our input shape. We've got out channels, which are our hidden units. We've got a kernel size, which equals three. Or this could be a tuple as well, three by three. But I just like to keep it as three. We've got a stride and we've got padding. Now, because these are values, we can set ourselves. What are they referred to as?
17:27:26
Let's write this down. Values, we can set ourselves in our neural networks.
17:27:32
In our nn's neural networks are called hyper parameters. So these are the hyper parameters of nn.com 2d. And you might be thinking, what is 2d for? Well, because we're working with two-dimensional data, our images have height and width. There's also com 1d for one-dimensional data, 3d for three-dimensional data. We're going to stick with 2d for now. And so what do each of these hyper parameters do? Well, before we go through what each one of them
17:28:02
do, we're going to do that when we step by step through this particular layer. What we've just done is we've replicated this particular layer of the CNN explainer website. We've still got the relu. We've still got another conv and a relu and a max pool and a conv and a relu and a conv and a relu and a max pool. But this is the block I was talking about. This is one block here
17:28:25
of this neural network, or at least that's how I've broken it down. And this is another block. You might notice that they're comprised of the same layers just stacked on top of each other. And then we're going to have an output layer. And if you want to learn about where the hyper parameters came from, what we just coded, where could you learn about those? Well, one, you could go, of course, to the PyTorch documentation, PyTorch, and then com 2d. You can read about it there.
17:28:53
There's the mathematical operation that we talked about or briefly stepped on before, or touched on, stepped on. Is that the right word? So create a conv layer. It's there. But also this is why I showed you this beautiful website so that you can read about these hyper parameters down here. Understanding hyper parameters. So your extra curriculum for this video is to go through this little graphic here and see if you can find out what padding means,
17:29:21
what the kernel size means, and what the stride means. I'm not going to read through this for you. You can have a look at this interactive plot. We're going to keep coding because that's what we're all about here. If and out, code it out. So we're going to now add a relu layer. And then after that, we're going to add another conv 2d layer. And the in channels here is going to be the hidden units, because we're going to take the output size of this layer and use it as
17:29:50
the input size to this layer. We're going to keep going here. Out channels equals hidden units again in this case. And then the kernel size is going to be three as well. Stride will be one. Padding will be one. Now, of course, we can change all of these values later on, but just bear with me while we set them how they are. We'll have another relu layer. And then we're going to finish off with a nn max pool 2d layer. Again, the 2d comes from the same reason we use comf2d. We're working
17:30:23
with 2d data here. And we're going to set the kernel size here to be equal to two. And of course, this can be a tuple as well. So it can be two two. Now, where could you find out about nn max pool 2d? Well, we go nn max pool 2d. What does this do? applies a 2d max pooling over an input signal composed of several input planes. So it's taking the max of an input. And we've got some parameters here, kernel size, the size of the window to take the max over. Now, where have we
17:30:55
seen a window before? I'm just going to close these. We come back up. Where did we see a window? Let's dive into the max pool layer. See where my mouse is? Do you see that two by two? Well, that's a window. Now, look at the difference between the input and the output. What's happening? Well, we have a tile that's two by two, a window of four. And the max, we're taking the max of that tile. In this case, it's zero. Let's find the actual value. There we go. So if you look at those
17:31:23
four numbers in the middle inside the max brackets, we have 0.07, 0.09, 0.06, 0.05. And the max of all those is 0.09. And you'll notice that the input and the output shapes are different. The output is half the size of the input. So that's what max pooling does, is it tries to take the max value of whatever its input is, and then outputs it on the right here. And so as our data, this is a trend in all of deep learning, actually. As our image moves through, this is what you'll
17:31:59
notice. Notice all the different shapes here. Even if you don't completely understand what's going on here, you'll notice that the two values here on the left start to get smaller and smaller as they go through the model. And what our model is trying to do here is take the input and learn a compressed representation through each of these layers. So it's going to smoosh and smoosh and smoosh trying to find the most generalizable patterns to get to the ideal output. And that
17:32:27
input is eventually going to be a feature vector to our final layer. So a lot going on there, but let's keep coding. What we've just completed is this first block. We've got a cons layer, a relu layer, a cons layer, a relu layer, and a max pool layer. Look at that, cons layer, relu layer, cons layer, relu layer, max pool. Should we move on to the next block? We can do this one a bit faster now because we've already coded the first one. So I'm going to do nn.sequential as
17:32:55
well. And then we're going to go nn.com2d. We're going to set the in channels. What should the in channels be here? Well, we're going to set it to hidden units as well because our network is going to flow just straight through all of these layers. And the output size of this is going to be hidden units. And so we want the in channels to match up with the previous layers out channels. So then we're going to go out channels equals hidden units as well. We're going to set the
17:33:28
kernel size, kernel size equals three, stride equals one, padding equals one, then what comes next? Well, because the two blocks are identical, the con block one and com two, we can just go the exact same combination of layers. And then relu and n.com2d in channels equals hidden units.
17:33:53
Out channels equals, you might already know this, hidden units. Then we have kernel size equals three, oh, 32, don't want it that big, stride equals one, padding equals one, and what comes next? Well, we have another relu layer, relu, and then what comes after that? We have another max pool. And then max pool 2d, kernel size equals two, beautiful. Now, what have we coded up so far? We've got this block, number one, that's what this one on the inside
17:34:27
here. And then we have com two, relu two, com two, relu two, max pool two. So we've built these two blocks. Now, what do we need to do? Well, we need an output layer. And so what did we do before when we made model one? We flattened the inputs of the final layer before we put them to the last linear layer. So flatten. So this is going to be the same kind of setup as our classifier layer. Now, I say that on purpose, because that's what you'll generally hear the last output layer
17:35:02
in a classification model called is a classifier layer. So we're going to have these two layers are going to be feature extractors. In other words, they're trying to learn the patterns that best represent our data. And this final layer is going to take those features and classify them into our target classes. Whatever our model thinks best suits those features, or whatever our model thinks those features that it learned represents in terms of our classes. So let's code it out.
17:35:29
We'll go down here. Let's build our classifier layer. This is our biggest neural network yet. You should be very proud. We have an end of sequential again. And we're going to pass in an end of flatten, because the output of these two blocks is going to be a multi-dimensional tensor, something similar to this size 131310. So we want to flatten the outputs into a single feature vector. And then we want to pass that feature vector to an nn.linear layer. And we're going to
17:36:05
go in features equals hidden units times something times something. Now, the reason I do this is because we're going to find something out later on, or time zero, just so it doesn't error. But sometimes calculating what you're in features needs to be is quite tricky. And I'm going to show you a trick that I use later on to figure it out. And then we have out features relates to our output shape, which will be the length of how many classes we have, right? One value for
17:36:35
each class that we have. And so with that being said, let's now that we've defined all of the components of our tiny VGG architecture. There is a lot going on, but this is the same methodology we've been using the whole time, defining some components, and then putting them together to compute in some way in a forward method. So forward self X. How are we going to do this? Are we going to set X is equal to self, comp block one X. So X is going to go through comp block one,
17:37:11
it's going to go through the comp 2D layer, relu layer, comp 2D layer, relu layer, max pool layer, which will be the equivalent of an image going through this layer, this layer, this layer, this layer, this layer, and then ending up here. So we'll set it to that. And then we can print out X dot shape to get its shape. We'll check this later on. Then we pass X through comp block two,
17:37:38
which is just going to go through all of the layers in this block, which is equivalent to the output of this layer going through all of these layers. And then because we've constructed a classifier layer, we're going to take the output of this block, which is going to be here, and we're going to pass it through our output layer, or what we've termed it, our classifier layer. I'll just print out X dot shape here, so we can track the shape as our model moves through the architecture.
17:38:04
X equals self dot classifier X. And then we're going to return X. Look at us go. We just built our first convolutional neural network by replicating what's on a CNN explainer website. Now, that is actually very common practice in machine learning is to find some sort of architecture that someone has found to work on some sort of problem and replicate it with code and see if it works on your own problem. You'll see this quite often. And so now let's instantiate a model.
17:38:42
Go torch dot manual C. We're going to instantiate our first convolutional neural network.
17:38:48
Model two equals fashion amnest. We will go model V two. And we are going to set the input shape. Now, what will the input shape be? Well, I'll come to the layer up here. The input shape is the number of channels in our images. So do we have an image ready to go image shape? This is the number of color channels in our image. We have one. If we had color images, we would set the input shape to three. So the difference between our convolutional neural network,
17:39:23
our CNN, tiny VGG, and the CNN explainer tiny VGG is that they are using color images. So their input is three here. So one for each color channel, red, green and blue. Whereas we have black and white images. So we have only one color channel. So we set the input shape to one. And then we're going to go hidden units equals 10, which is exactly the same as what tiny VGG has used. 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. So that sets the hidden units value of each of our
17:39:57
layers. That's the power of creating an initializer with hidden units. And then finally, our output shape is going to be what we've seen this before. This is going to be the length of our class names, one value for each class in our data set. And of course, we're going to send this model to the device. We're going to hit shift and enter. Oh, no, what did we get wrong? Out channels, output shape. Where did I spell wrong? Out channels, out channels, out channels. I forgot an L.
17:40:28
Of course, typo. Oh, kernel size and other typo. Did you notice that? Kernel size, kernel size, kernel size, kernel size. Where did we spell this wrong? Oh, here.
17:40:44
Kernel size. Are there any other typos? Probably.
17:40:50
A beautiful. There we go. Okay, what have we got? Initializing zero, obtenses and non-op. Oh, so we've got an issue here and error here because I've got this. But this is just to, there's a trick to calculating this. We're going to cover this in another video. But pay yourself on the back. We've written a fair bit of code here. This is a convolutional neural network that replicates the tiny VGG architecture on the CNN explainer website. Now, don't forget,
17:41:19
your extra curriculum is to go through this website for at least 20 minutes and read about what's happening in our models. We're focused on code here. But this is particularly where you want to pay attention to. If you read through this understanding hyper parameters and play around with this, the next couple of videos will make a lot more sense. So read about padding, read about kernel size and read about stride. I'll see you in the next video. We're going to go
17:41:43
through our network step by step. Welcome back. Now, I'm super stoked because in the last video, we coded together our first ever convolutional neural network in PyTorch. So well done. We replicated the tiny VGG architecture from the CNN explainer website, my favorite place for learning about CNNs in the browser. So now we introduced two new layers that we haven't seen before, conv2d and maxpool2d. But they all have the same sort of premise of what we've been doing so far
17:42:17
is that they're trying to learn the best features to represent our data in some way, shape or form. Now, in the case of maxpool2d, it doesn't actually have any learnable parameters. It just takes the max, but we're going to step through that later on. Let's use this video to step through and then conv2d. We're going to do that with code. So I'll make a new heading here. 7.1
17:42:43
stepping through and then conv2d. Beautiful. Now, where could we find out what's going on in an end comp2d? Well, of course, we have the documentation and then comp2d. We've got PyTorch. So if you want to learn the mathematical operation that's happening, we have this value here, this operation here. Essentially, it's saying the output is equal to the bias term times something, plus the sum of the weight times something times the input. So do you see how just the weight
17:43:18
matrix, the weight tensor and the bias value, manipulating our input in some way equals the output? Now, if we map this, we've got batch size, channels in, height, width, channels out, out, out, et cetera, et cetera. But we're not going to focus too much on this. If you'd like to read more into that, you can. Let's try it with code. And we're going to reproduce this particular layer here, the first layer of the CNN explainer website. And we're going to do it with a dummy input.
17:43:50
In fact, that's one of my favorite ways to test things. So I'm just going to link here the documentation.
17:43:58
See the documentation for an end comp2d here. And if you'd like to read through more of this, of course, this is a beautiful place to learn about what's going on. There's the shape how to calculate the shape, height out, width out, et cetera. That's very helpful if you need to calculate input and output shapes. But I'll show you my trick for doing so later on. We have here, let's create some dummy data. So I'm going to set torch manual seed. We
17:44:30
need it to be the same size as our CNN explainer data. So 64, 64, 3. But we're going to do it pie torch style. This is color channels last. We're going to do color channels first. So how about we create a batch of images, we're going to be writing torch dot rand n. And we're going to pass in size equals 32, three, 64, 64. And then we're going to create a singular image by taking the first of that. So image is zero. Now, let's get the image batch shape. Because a lot of
17:45:09
machine learning, as I've said before, and deep learning is making sure your data has the right shape. So let's check images dot shape. And let's check single image shape. We're going to go test image dot shape. And finally, we're going to print, what does the test image look like?
17:45:34
We'll get this on a new line, hey, new line test image, this is of course not going to be an actual image is just going to be a collection of random numbers. And of course, that is what our model is currently comprised of model two, if we have a look at what's on the insides, we are going to see a whole bunch of random numbers. Look at all this. What do we have? We scroll up is going to give us a name for something. We have comp block two, two, we have a weight,
17:46:02
we have a bias, keep going up, we go right to the top, we have another weight, keep going down, we have a bias, a weight, et cetera, et cetera. Now, our model is comprised of random numbers, and what we are trying to do is just like all of our other models is pass data in and adjust the random numbers within these layers to best represent our data. So let's see what happens if we pass some random data through one of our comp2d layers. So let's go here, we're going to
17:46:33
create a single comp2d layer. So comp layer equals, what is it equal? And then comp2d, and we're going to set the in channels is equal to what? Oh, revealed the answer too quickly. Three. Why is it three? Well, it's because the in channels is the same number of color channels as our images. So if we have a look at our test image shape, what do we have? Three, it has three color channels. That is the same as the value here, except the order is reversed. This is color
17:47:14
channels last, pytorch defaults to color channels first. So, or for now it does, in the future this may change. So just keep that in mind. So out channels equals 10. This is equivalent to the number of hidden units we have. One. Oh, I don't want that one just yet. One, two, three, four, five, six, seven, eight, nine, 10. So we have that 10 there. So we have 10 there. And then we have kernel size. Oh, what is the kernel? Well, it's not KFC. I can tell you that. And then we have
17:47:44
stride. And then we have padding. We're going to step through these in a second. But let's check out the kernel. And this kernel can also be three by three. But it's a shortcut to just type in three. So that's what it actually means. If you just type in a single number, it's equivalent to typing in a tuple. Now, of course, you could find that out by reading through the documentation here. But where did I get that value? Well, let's dive into this beautiful website. And let's see what
17:48:09
happening. So we have a kernel here, which is also called a filter. So the thing I'm talking about is this little square here, this kernel. Oh, we can see the weights there at the top. This is how beautiful this website is. So if we go over there, this is what's going to happen. This is a convolution. It starts with this little square, and it moves pixel by pixel across our image. And you'll notice that the output is creating some sort of number there. And you'll notice in the middle, we have a
17:48:41
mathematical operation. This operation here is what's happening here. I wait times the input.
17:48:50
That's what we've got there. Now, the beauty of PyTorch is it does all of this behind the scenes for us. So again, if you'd like to dig more into the mathematical operation behind the scenes, you've got the resource here. And you've also got plenty of other resources online. We're going to focus on code for now. So if we keep doing this across our entire image, we get this output over here. So that's the kernel. And now where did I get three by three from? Well, look at
17:49:14
this. One, two, three, one, two, three, one, two, three, three by three, we have nine squares. Now, if we scroll down, this was your extracurricular for the last video, understanding hyperparameters. What happens if we change the kernel size to three by three? Have a look at the red square on the left. Now, if we change it to two by two, it changed again. Three by three. This is our kernel, or also known as a filter, passing across our image, performing some sort of mathematical
17:49:43
operation. And now the whole idea of a convolutional layer is to try and make sure that this kernel performs the right operation to get the right output over here. Now, what do these kernels learn? Well, that is entirely up to the model. That's the beauty of deep learning is that it learns how to best represent our data, hopefully, on its own by looking at more data. And then so if we jump back in here, so that's the equivalent of setting kernel size three by three. What if
17:50:13
we set the stride equal to one? Have we got this in the right order? It doesn't really matter. Let's go through stride next. If we go to here, what does stride say? Stride of the convolution of the convolving kernel. The default is one. Wonderful. Now, if we set the stride, or if we keep it at one, it's a default one, it's going to hop over, watch the red square on the left. It's going to hop over one pixel at a time. So the convolution, the convolving, happens one
17:50:43
pixel at a time. That's what the stride sets. Now, watch what happens when I change the stride value to the output shape. Wow. Do you notice that it went down? So we have here, the kernel size is still the same. But now we're jumping over two pixels at a time. Notice how on the left, two pixels become available. And then if I jump over again, two pixels. So the reason why the output compresses is because we're skipping some pixels as we go across the image. And now this
17:51:14
pattern happens throughout the entire network. That's one of the reasons why you see the size of our input or the size of each layer go down over time. What our convolutional layer is doing, and in fact, a lot of deep learning neural networks do, is they try to compress the input into some representation that best suits the data. Because it would be no point of just memorizing the exact patterns, you want to compress it in some way. Otherwise, you just might as well move
17:51:47
your input data around. You want to learn generalizable patterns that you can move around. And so we keep going. We've got padding equals zero. Let's see what happens here. If we change the padding value, what happens? Up, down. Notice the size here. Oh, we've added two extra pixels around the edge. Now if we go down, one extra pixel. Now if we go zero, now why might we do that?
17:52:13
If we add some padding on the end, well, that's so that our kernel can operate on what's going on here in the corner. In case there's some information on the edges of our image. Then you might be thinking, Daniel, there's a whole bunch of values here. How do we know what to set them? Well, you notice that I've just copied exactly what is going on here.
17:52:33
There's a three by three kernel. There's no padding on the image. And the stride is just going one by one. And so that's often very common in machine learning, is that when you're just getting started and you're not sure what values to set these values to, you just copy some existing values from somewhere and see if it works on your own problem. And then if it doesn't, well, you can adjust them. So let's see what happens when we do that. So pass the data through
17:53:02
the convolutional layer. So let's see what happens. Conv output equals conv layer. Let's pass it our test image. And we'll check the conv output. What happens?
17:53:17
Oh no, we get an error. Of course we get a shape error. One of the most common issues of machine learning and deep learning. So this is saying that our input for the conv layer expects a four dimensional tensor, except it got a three dimensional input size of 364 64. Now, how do we add an extra dimension to our test image? Let's have a look. How would we add a batch dimension over on the left here? We can go unsqueeze zero. So now we have a four dimensional tensor. Now, just keep
17:53:53
in mind that if you're running this layer and then com2d on a pytorch version, that is, I believe they fixed this or they changed it in pytorch. What am I on? I think this Google collab instance is on 1.10. I think you might not get this error if you're running 1.11. So just keep that in mind. Like this should work if you're running 1.11. But if it doesn't, you can always unsqueeze here.
17:54:22
And let's see what happens. Look at that. We get another tensor output. Again, this is just all random numbers though, because our test image is just random numbers. And our conv layer is instantiated with random numbers. But we'll set the manual seed here. Now, if our numbers are different to what's, if your numbers are different to what's on my screen, don't worry too much. Why is that? Because our conv layer is instantiated with random numbers. And our test
17:54:48
image is just random numbers as well. What we're paying attention to is the input and output shapes.
17:54:55
Do you see what just happened? We put our input image in there with three channels. And now because we've set out channels to be 10, we've got 10. And we've got 62, 62. And this is just the batch size. It just means one image. So essentially our random numbers, our test image, have gone through the convolutional layer that we created, have gone through this mathematical operation with regards to all the values that we've set, we've put the weight tensor, well,
17:55:19
actually PyTorch created that for us. PyTorch has done this whole operation for us. Thank you, PyTorch. It's gone through all of these steps across. You could code this all by hand if you want, but it's a lot easier and simpler to use a PyTorch layer. And it's done this. And now it's created this output. Now, whatever this output is, I don't know, it is random numbers, but this same process will happen if we use actual data as well. So let's see what happens if we change
17:55:47
the values kernel size we increase. Notice how our output has gotten smaller because we're using a bigger kernel to convolve across the image. What if we put this to three, three back to what it was and stride of two? What do you think will happen? Well, our output size basically halves because we're skipping two pixels at a time. We'll put that back to one. What do you think will happen if we set padding to one? 64, 64. We get basically the same size because we've added an
17:56:16
extra pixel around the edges. So you can play around with this. And in fact, I encourage you to do this is what we just did. Padding one, we just added an extra dummy zero pixel around the edges. So practice with this, see what happens as you pass our test image, random numbers, through a conv 2d layer with different values here. What do you think will happen if you change this to 64? Give that a shot and I'll see you in the next video. Who's ready to step through
17:56:48
the nn max pool 2d layer? Put your hand up. I've got my hand up. So let's do it together, hey, we've got 7.2. Now you might have already given this a shot yourself. Stepping through
17:57:03
nn max pool 2d. And this is this is what I do for a lot of different concepts that I haven't gone through before is I just write some test code and see what the inputs and outputs are. And so where could we find out about max pool 2d? Well, of course, we've got the documentation. I'm just going to link this in here. Max pool 2d. In the simplest case, the output value of layer with input size nchw output nch out w out. By the way, this is number of batches,
17:57:38
color channels, height, width. And this is the output of that layer. And kernel size, which is a parameter up here, k h k w can be precisely described as out is going to be the max of some value, depending on the kernel size and the stride. So let's have a look at that in practice. And of course, you can read further through the documentation here. I'll just grab the link for this actually. So it's here. Wonderful. And let's now first try it with our test image that we
17:58:13
created above. So just highlight what the test image is. A bunch of random numbers in the same shape as what a single image would be if we were to replicate the image size of the CNN explainer.
17:58:28
By the way, we'll have a look at a visual in a second of max pool here. But you can go through that on your own time. Let's if in doubt, code it out. So we're going to print out the original image shape without unsqueezed dimension. Because recall that we had to add an extra dimension to pass it through our com2d layer. Now, if you're using a later version of PyTorch, you might not get an error if you only use a three dimensional image tensor and pass it through a comp layer.
17:58:59
So we're going to pass it in test image, original shape, test image dot shape. So this is just going to tell us what the line of code in the cell above tells us. But that's fine. I like to make pretty printouts, you know, test image with unsqueezed dimension. So this is just going to be our test image. And we're going to see what happens when we unsqueeze a dimension, unsqueeze on zero for dimension. That is about to say first, but it's the zero. Now we're going to create a sample
17:59:34
nn max pool 2d layer. Because remember, even layers themselves in torch dot nn are models of their own accord. So we can just create a single, this is like creating a single layer model here. We'll set the kernel size equal to two. And recall, if we go back to CNN explainer, kernel size equal to two results in a two by two square, a two by two kernel that's going to convolve over our image, like so. And this is an example input, an example output. And you can see
18:00:09
the operation that max pooling does here. So just keep that in mind as we pass some sample data through our max pool layer. And now let's pass data through it. I actually will pass it through just the conv layer first, through just the conv layer. Because that's sort of how you might stack things, you might put a convolutional layer and then a max pool layer on top of that convolutional layer. So test image through conv. We'll create a variable here, equals our conv layer.
18:00:41
Is going to take as an input, our test image dot unsqueeze on the zero dimension again.
18:00:50
Beautiful. Now we're going to print out the shape here. This is just highlighting how I like to troubleshoot things is I do one step, print the shape, one step, print the shape, see what is happening as our data moves through various layers. So test image through conv.shape, we'll see what our conv layer does to the shape of our data. And then we're going to pass data through
18:01:19
max pool layer, which is the layer we created a couple of lines above this one here. So let's see what happens. Test image through current type at the moment through conv and max pool. So quite a long variable name here, but this is to help us avoid confusion of what's going on. So we go test image through conv. So you notice how we're taking the output of our convolutional layer, this here, and we're passing it through our max pool layer, which has another
18:01:53
typo. Wonderful. And finally, we'll print out the shape, shape after going through conv layer
18:02:04
and max pool layer. What happens here? So we want test image through conv and max pool. Let's see how our max pool layer manipulates our test images shape. You ready? Three, two, one, let's go. What do we get? Okay. So we have the test image original shape, recall that our test image is just a collection of random numbers. And of course, our conv layer is going to be instantiated with random numbers. And max pool actually has no parameters. It just
18:02:39
takes the maximum of a certain range of inner tensor. So when we unsqueeze the test image as the input, we get an extra dimension here. When we pass it through our conv layer. Oh, where did this 64 come from? 164 64 64 64. Let's go back up to our conv layer. Do you notice how that we get the 64 there because we changed the out channels value? If we change this back to 10, like what's in the CNN explainer model? One, two, three, four, five, six, seven, eight, nine, 10. What do you think will
18:03:17
happen there? Well, we get a little highlight here. 10. Then we keep going. I'll just get rid of this extra cell. We don't need to check the version anymore. We'll check the test image shapes still three 64 64. But then as we pass it through the conv layer here, we get a different size now. So it originally had three channels as the input for color channels, but we've upscaled it to 10 so that we have 10 hidden units in our layer. And then we have 64 64. Now, again,
18:03:51
these shapes will change if we change the values of what's going on here. So we might put padding to zero. What happens there? Instead of 64 64, we get 62 62. And then what happens after we pass it through the conv layer and then through the max pool layer? We've got 110 64 64. And now we have 110 32 32. Now, why is that? Well, let's go back into the CNN explainer, jump into this max pool layer here. Maybe this one because it's got a bit more going on. Do you notice on the left here is
18:04:27
the input? And we've got a two by two kernel here. And so the max pooling layer, what it does is it takes the maximum of whatever the input is. So you'll notice the input is 60 60 in this case. Whereas the output over here is 30 30. Now, why is that? Well, because the max operation here is reducing it from section of four numbers. So let's get one with a few different numbers.
18:04:55
There we go. That'll do. So it's taking it from four numbers and finding the maximum value within those four numbers here. Now, why would it do that? So as we've discussed before, what deep learning neural network is trying to do or in this case, a CNN is take some input data and figure out what features best represent whatever the input data is and compress them into a feature vector that is going to be our output. Now, the reason being for that is because you could consider it
18:05:28
from a neural networks perspective is that intelligence is compression. So you're trying to compress the patterns that make up actual data into a smaller vector space, go from a higher dimensional space to a smaller vector space in terms of dimensionality of a tensor. But still, this smaller dimensionality space represents the original data and can be used to predict on future data. So that's the idea behind Max Paul is, hey, if we've got these learned features from our
18:05:59
convolutional layers, will the patterns, will the most important patterns stay around if we just take the maximum of a certain section? So do you notice how the input here, we still have, you can still see the outline of the car here, albeit a little bit more pixelated, but just by taking the max of a certain region, we've got potentially the most important feature of that little section. And now, of course, you could customize this value here. If when we
18:06:27
create our max pool layer, you could increase the kernel size to four by four. What do you think will happen if we can increase it to four? So here, we've got a two by two kernel. If we increase it to four by four, what happens? Ah, do you notice that we've gone from 62 to 15, we've essentially divided our feature space by four, we've compressed it even further. Now, will that work? Well, I'm not sure. That's part of the experimental nature of machine learning, but we're going to
18:06:57
keep it at two for now. And so this is with our tensor here 6464. But now let's do the same as what we've done above, but we'll do it with a smaller tensor so that we can really visualize things. And we're going to just replicate the same operation that's going on here. So let's go here, we'll create another random tensor. We'll set up the manual seed first. And we're going to create a random tensor with a similar number of dimensions. Now, recall dimensions don't tell you, so this
18:07:35
is a dimension 1364 64. That is a dimension. The dimensions can have different values within themselves. So we want to create a four dimensional tensor to our images. So what that means is, let me just show you it's way easy to explain things when we've got code is torch dot rand n. And we're going to set it up as size equals one, one, two, two. We can have a look at this random tensor. It's got four dimensions. One, two, three, four. So you could have a batch size,
18:08:12
color channels, and height width, a very small image, but it's a random image here. But this is quite similar to what we've got going on here, right? Four numbers. Now, what do you think will happen if we create a max pool layer, just like we've done above, create a max pool layer. So we go max pool layer, just repeating the code that we have in the cell above, that's all right, a little bit of practice. Kernel size equals two. And then we're going to pass the random tensor
18:08:46
through the max pool layer. So we'll go max pool tensor equals max pool layer. And we're going to pass it in the random tensor. Wonderful. And then we can print out some shapes and print out some tenses. As we always do to visualize, visualize, visualize. So we're going to write in here max pool tensor on a new line. We'll get in the max pool tensor. We'll see what this looks like. And we'll also print out max pool tensor shape. And we can probably print out random tensor
18:09:32
itself, as well as its shape as well. We'll get the shape here, dot shape. And we'll do the same for the random tensor. So print, get a new line, random tensor, new line, random tensor. And then we'll get the shape. Random tensor shape, random tensor. Oh, a lot of coding here. That's, that's the fun part about machine learning, right? You get to write lots of code. Okay. So we're visualizing what's going on with our random tensor. This is what's happening within the max pool layer.
18:10:11
We've seen this from a few different angles now. So we have a random tensor of numbers, and we've got a size here. But the max pool tensor, once we pass our random tensor, through the max pool layer, what happens? Well, we have 0.3367, 1288, 2345, 2303. Now, what's the max of all these? Well, it takes the max here is 3367. Oh, and we've got the random tensor down there. We don't want that. And see how we've reduced the shape from two by two to one
18:10:44
by one. Now, what's going on here? Just for one last time to reiterate, the convolutional layer is trying to learn the most important features within an image. So if we jump into here, now, what are they? Well, we don't decide what a convolutional layer learns. It learns these features on its own. So the convolutional layer learns those features. We pass them through a relu nonlinear activation in case our data requires nonlinear functions. And then we pass
18:11:18
those learned features through a max pool layer to compress them even further. So the convolutional layer can compress the features into a smaller space. But the max pooling layer really compresses them. So that's the entire idea. One more time, we start with some input data. We design a neural network, in this case, a convolutional neural network, to learn a compressed representation of what our input data is, so that we can use this compressed representation to later on make
18:11:46
predictions on images of our own. And in fact, you can try that out if you wanted to click here and add your own image. So I'd give that a go. That's your extension for this video. But now we've stepped through the max pool 2D layer and the conv 2D layer. I think it's time we started to try and use our tiny VGG network. This is your challenge is to create a dummy tensor and pass it through this model. Pass it through its forward layer and see what happens to the shape of your dummy tensor
18:12:21
as it moves through conv block 1 and conv block 2. And I'll show you my trick to calculating the in features here for this final layer, which is equivalent to this final layer here.
18:12:34
I'll see you in the next video.
18:12:37
Over the last few videos, we've been replicating the tiny VGG architecture from the CNN explainer website. And I hope you know that this is this actually quite exciting because years ago, this would have taken months of work. And we've just covered we've broken it down over the last few videos and rebuilt it ourselves with a few lines of PyTorch code.
18:12:58
So that's just goes to show how powerful PyTorch is and how far the deep learning field has come. But we're not finished yet. Let's just go over to our keynote. This is what we've done. CNN explainer model. We have an input layer. We've created that. We have com2d layers. We've created those. We have relo activation layers. We've created those. And finally, we have pulling layers. And then we finish off with an output layer.
18:13:30
But now let's see what happens when we actually pass some data through this entire model. And as I've said before, this is actually quite a common practice is you replicate a model that you found somewhere and then test it out with your own data. So we're going to start off by using some dummy data to make sure that our model works. And then we're going to pass through. Oh, I've got another slide for this. By the way, here's a breakdown of torch and
18:13:58
N com2d. If you'd like to see it in text form, nothing here that we really haven't discussed before, but this will be in the slides if you would like to see it. Then we have a video animation. We've seen this before, though. And plus, I'd rather you go through the CNN explainer website on your own and explore this different values rather than me just keep talking about it. Here's what we're working towards doing. We have a fashion MNIST data set. And we have
18:14:24
our inputs. We're going to numerically encode them. We've done that already. Then we have our convolutional neural network, which is a combination of convolutional layers, nonlinear activation layers, pooling layers. But again, these could be comprised in many different ways, shapes and forms. In our case, we've just replicated the tiny VGG architecture. And then finally, we want to have an output layer to predict what class of clothing a particular input image is.
18:14:52
And so let's go back. We have our CNN model here. And we've got model two. So let's just practice a dummy forward pass here. We're going to come back up a bit to where we were. We'll make sure we've got model two. And we get an error here because I've times this by zero. So I'm going to just remove that and keep it there. Let's see what happens if we create a dummy tensor and pass it through here. Now, if you recall what our image is, do we have image? This is a fashion MNIST
18:15:29
image. So I wonder if we can go plot dot M not M show image. And I'm going to squeeze that. And I'm going to set the C map equal to gray. So this is our current image. Wonderful. So there's our current image. So let's create a tensor. Or maybe we just try to pass this through the model and see what happens. How about we try that model image? All right, we're going to try the first pass forward pass. So pass image through model. What's going to happen? Well, we get an
18:16:12
error. Another shape mismatch. We've seen this before. How do we deal with this? Because what is the shape of our current image? 128, 28. Now, if you don't have this image instantiated, you might have to go back up a few cells. Where did we create image? I'll just find this. So just we created this a fairly long time ago. So I'm going to probably recreate it down the bottom. My goodness, we've written a lot of code. Well, don't do us. We could create a dummy tensor
18:16:45
if we wanted to. How about we do that? And then if you want to find, oh, right back up here, we have an image. How about we do that? We can just do it with a dummy tensor. That's fine.
18:16:58
We can create one of the same size. But if you have image instantiated, you can try that out. So there's an image. Let's now create an image that is, or a random tensor, that is the same shape as our image. So rand image tensor equals what torch dot rand n. And we're going to pass in size equals 128, 28. Then if we get rand image tensor,
18:17:32
we check its shape. What do we get? So the same shape as our test image here, but it's just going to be random numbers. But that's okay. We just want to highlight a point here of input and output shapes. We want to make sure our model works. Can our random image tensor go all the way through our model? That's what we want to find out. So we get an error here. We have four dimensions, but our image is three dimensions. How do we add an extra dimension
18:17:54
for batch size? Now you might not get this error if you're running a later version of pie torch. Just keep that in mind. So unsqueeze zero. Oh, expected all tensors to be on the same device, but found at least two devices. Again, we're going through all the three major issues in deep learning. Shape mismatch, device mismatch, data type mismatch. So let's put this on the device, two target device, because we've set up device agnostic code.
18:18:21
That one and that two shapes cannot be multiplied. Oh, but we can output here. That is very exciting. So what I might do is move this a couple of cells up so that we can tell what's going on. I'm going to delete this cell. So where do these shapes come from?
18:18:42
Well, we printed out the shapes there. And so this is what's happened when our, I'll just create our random tensor. I'll bring our random tensor up a bit too. Let's bring this up.
18:18:53
There we go. So we pass our random to image tensor through our model, and we've made sure it's got four dimensions by unsqueeze zero. And we make sure it's on the same device as our model, because our model has been sent to the GPU. And this is what happens as we pass our random image tensor. We've got 12828 instead of previously we've seen 6464.3, which is going to clean this up a bit. And we get different shapes here. So you'll notice that as our input, if it was 6464.3
18:19:27
goes through these layers, it gets shaped into different values. Now this is going to be universal across all of the different data sets you work on, you will be working with different shapes. So it's important to, and also quite fun, to troubleshoot what shapes you need to use for your different layers. So this is where my trick comes in. To find out the shapes for different layers, I often construct my models, how we've done here, as best I can with the information
18:19:53
that I've got, such as replicating what's here. But I don't really know what the output shape is going to be before it goes into this final layer. And so I recreate the model as best I can. And then I pass data through it in the form of a dummy tensor in the same shape as my actual data. So we could customize this to be any shape that we wanted. And then I print the shapes of what's happening through each of the forward past steps. And so if we pass it through
18:20:21
this random tensor through the first column block, it goes through these layers here. And then it outputs a tensor with this size. So we've got 10, because that's how many output channels we've set. And then 14, 14, because our 2828 tensor has gone through a max pool 2d layer and gone through a convolutional layer. And then it goes through the next block, column block two, which is because we've put it in the forward method here. And then it outputs the shape. And if we go back down,
18:20:53
we have now a shape of one 10, seven, seven. So our previous tensor, the output of column block one, has gone from 1414 to seven seven. So it's been compressed. So let me just write this down here, output shape of column block one, just so we get a little bit more information.
18:21:15
And I'm just going to copy this, put it in here, that will become block two. And then finally, I want to know if I get an output shape of classifier. So if I rerun all of this, I don't get an output shape of classifier. So my model is running into trouble. Once it gets to, so I get the output of conv block one, I don't get an output of classifier. So this is telling me that I have an issue with my classifier layer. Now I know this, but I'm
18:21:51
not. Now I know this because, well, I've coded this model before, and the in features here, we need a special calculation. So what is going on with our shapes?
18:22:02
Mat one and mat two shapes cannot be multiplied. So do you see here, what is the rule of matrix multiplication? The inner dimensions here have to match. We've got 490. Where could that number have come from? And we've got 10 times 10. Now, okay, I know I've set hidden units to 10. So maybe that's where that 10 came from. And what is the output layer of the output shape of conv block two? So if we look, we've got the output shape of conv block two. Where does that go?
18:22:38
The output of conv block two goes into our classifier model. And then it gets flattened. So that's telling us something there. And then our NN linear layer is expecting the output of the flatten layer as it's in features. So this is where my trick comes into play. I pass the output of conv block two into the classifier layer. It gets flattened. And then that's what my NN not linear layer is expecting. So what happens if we flatten this shape here? Do we get
18:23:16
this value? Let's have a look. So if we go 10 times seven times seven, 490. Now, where was this 10? Well, that's our hidden units. And where were these sevens? Well, these sevens are the output of conv block two. So that's my trick. I print the shapes of previous layers and see whether or not they line up with subsequent layers. So if we go time seven times seven, we're going to have hidden units equals 10 times seven times seven. Where do we get the two sevens? Because that is
18:23:58
the output shape of conv block two. Do you see how this can be a little bit hard to calculate ahead of time? Now, you could calculate this by hand if you went into n conv 2d. But I prefer to write code to calculate things for me. You can calculate that value by hand. If you go through, H out W out, you can add together all of the different parameters and multiply them and divide them and whatnot. You can calculate the input and output shapes of your convolutional layers.
18:24:28
You're more than welcome to try that out by hand. But I prefer to code it out. If and out code it out. Now, let's see what happens if we run our random image tensor through our model. Now, do you think it will work? Well, let's find out. All we've done is we've added this little line here, times seven times seven. And we've calculated that because we've gone, huh, what if we pass a tensor of this dimension through a flattened layer? And what is our rule of matrix multiplication?
18:25:00
The inner dimensions here must match. And why do we know that these are matrices? Well, mat one and mat two shapes cannot be multiplied. And we know that inside a linear layer is a matrix multiplication. So let's now give this a go. We'll see if it works.
18:25:22
Oh, ho ho. Would you look at that? That is so exciting. We have the output shape of the classifier is one and 10. We have a look, we have one number one, two, three, four, five, six, seven, eight, nine, 10, one number for each class in our data set. Wow. Just like the CNN explain a website, we have 10 outputs here. We just happen to have 10 classes as well. Now, this number again could be whatever you want. It could be 100, could be 30, could be three, depending on how many classes
18:25:55
you have. But we have just figured out the input and output shapes of each layer in our model. So that's very exciting. I think it's now time we've passed a random tensor through. How about we pass some actual data through our model? In the next video, let's use our train and test step functions to train our first convolutional neural network. I'll see you there.
18:26:24
Well, let's get ready to train our first CNN. So what do we need? Where are we up to in the workflow? Well, we've built a model and we've stepped through it. We know what's going on, but let's really see what's going on by training this CNN or see if it trains because we don't always know if it will on our own data set, which is of fashion MNIST. So we're going to set up a loss function and optimizer for model two. And just as we've done before, model two, turn that
18:26:54
into markdown. I'll just show you the workflow again. So this is what we're doing. We've got some inputs. We've got a numerical encoding. We've built this architecture and hopefully it helps us learn or it helps us make a predictive model that we can input images such as grayscale images of clothing and predict. And if we look where we are at the PyTorch workflow, we've got our data ready. We've built our next model. Now here's where we're up to picking a loss function and an optimizer.
18:27:29
So let's do that, hey, loss function, or we can do evaluation metrics as well. So set up loss function slash eval metrics slash optimizer. And we want from helper functions, import accuracy function, we don't need to reimport it, but we're going to do it anyway for completeness. Loss function equals nn dot cross entropy loss, because we are working with a multi class classification problem. And the optimizer, we're going to keep the same as what we've used before, torch dot
18:28:03
opt in SGD. And we'll pass it in this time, the params that we're trying to optimize are the parameters of model two parameters. And we'll use a learning rate of 0.1. Run that. And just to reiterate, here's what we're trying to optimize model two state dig. We have a lot of random weights in model two. Have a look at all this. There's the bias, there's the weight. We're going to try and optimize these to help us predict on our fashion MNIST data set. So without any further
18:28:37
ado, let's in the next video, go to the workflow, we're going to build our training loop. But thanks to us before, we've now got functions to do this for us. So if you want to give this a go, use our train step and test step function to train model two. Try that out. And we'll do it together in the next video. We're getting so close to training our model. Let's write some code to train our first thing in that model. Training and testing, I'm just going to make another heading
18:29:11
here. Model two, using our training and test functions. So we don't have to rewrite all of the steps in a training loop and a testing loop, because we've already created that functionality before through our train step function. There we go. Performs the training, or this should be performs a training step with model trying to learn on data loader. So let's set this up. We're going to set up torch manual seed 42, and we can set up a CUDA manual seed as well.
18:29:46
Just to try and make our experiments as reproducible as possible, because we're going to be using CUDA, we're going to measure the time because we want to compare our models, not only their performance in evaluation metrics, but how long they take to train from time it, because there's no point having a model that performs really, really well, but takes 10 times longer to train. Well, maybe there is, depending on what you're working on. Model two equals timer,
18:30:19
and we're going to train and test model, but the time is just something to be aware of, is that usually a better performing model will take longer to train. Not always the case, but just something to keep in mind. So for epoch in, we're going to use TQDM to measure the progress. We're going to create a range of epochs. We're just going to train for three epochs, keeping our experiment short for now, just to see how they work, epoch, and we're going to
18:30:48
print a new line here. So for an epoch in a range, we're going to do the training step, which is our train step function. The model is going to be equal to model two, which is our convolutional neural network, our tiny VGG. The data loader is just going to be equal to the train data loader, the same one we've used before. The loss function is going to be equal to the loss function that we've set up above, loss FN. The optimizer as well is going to be
18:31:17
the optimizer in our case, stochastic gradient descent, optimizer equals optimizer, then we set up the accuracy function, which is going to be equal to our accuracy function, and the device is going to be the target device. How easy was that? Now we do the same for the train or the testing step, sorry, the model is going to be equal to model two, and then the data loader is going to be the test data loader, and then the loss function is going to be our same
18:31:51
our same loss function. And then we have no optimizer for this, we're just going to pass in the accuracy function here. And then of course, the device is going to be equal to the device. And then what do we do now? Well, we can measure the end time so that we know how long the code here took to run. So let's go train time end for model two. This will be on the GPU, by the way, but this time it's using a convolutional neural network. And the total train time,
18:32:25
total train time for model two is going to be equal to print train time, our function that we created before as well, to help us measure start and end time. So we're going to pass in train to time start model two, and then end is going to be train time end model two. And then we're going to print out the device that it's using as well. So you're ready? Are you ready to train our first convolutional neural network? Hopefully this code works. We've created these functions before,
18:32:58
so it should be all right. But if and out, code it out, if and out, run the code, let's see what happens. Oh my goodness. Oh, of course. Oh, we forgot to comment out the output shapes. So we get a whole bunch of outputs for our model, because what have we done? Back up here, we forgot to. So this means every time our data goes through the forward pass, it's going to be printing out the output shapes. So let's just comment out these. And I think this cell is going
18:33:33
to take quite a long time to run because it's got so many printouts. Yeah, see, streaming output truncated to the last 5,000 lines. So we're going to try and stop that. Okay, there we go. Beautiful. That actually worked. Sometimes it doesn't stop so quickly. So we're going to rerun our fashion MSV to model cell so that we comment out these print lines. And then we'll just rerun these cells down here. Just go back through fingers crossed, there's no errors. And we'll train our
18:34:07
model again. Beautiful. Not as many printouts this time. So here we go. Our first CNN is training. How do you think it'll go? Well, that's what we have printouts, right? So we can see the progress. So you can see here all the functions that are being called behind the scenes from PyTorch. So thank you to PyTorch for that. There's our, oh, our train step function was in there. Train step. Wonderful. Beautiful. So there's epoch zero. Oh, we get a pretty good test accuracy.
18:34:35
How good is that? Test accuracy is climbing as well. Have we beaten our baseline? We're looking at about 14 seconds per epoch here. And then the final epoch. What do we finish at? Oh, 88.5. Wow. In 41.979 or 42 there about seconds. Again, your mileage may vary. Don't worry too much if these numbers aren't exactly the same on your screen and same with the training time because we might be using slightly different hardware. What GPU do I have today? I have a Tesla P100 GPU. You might
18:35:15
not have the same GPU. So the training time, if this training time is something like 10 times higher, you might want to look into what's going on. And if these values are like 10% lower or 10% higher, you might want to see what's going on with your code as well. But let's now calculate our Model 2 results. I think it is the best performing model that we have so far. Let's get a results dictionary. Model 2 results is so exciting. We're learning the power of convolutional neural
18:35:44
networks. Model 2 results equals a vowel model. And this is a function that we've created before.
18:35:52
So returns a dictionary containing the results of a model predicting on data loader. So now let's pass in the model, which will be our trained model to, and then we'll pass in the data loader, which will be our test data loader. And then, oops, excuse me, typo, our loss function will be, of course, our loss function. And the accuracy function will be accuracy function. And the device is already set, but we can reset anyway, device equals device. And we'll check
18:36:23
out the Model 2 results. Make some predictions. Oh, look at that. Model accuracy 88. Does that beat our baseline? Model 0 results. Oh, we did beat our baseline with a convolutional neural network. All right. So I feel like that's, uh, that's quite exciting. But now let's keep going on. And, uh, let's start to compare the results of all of our models. I'll see you in the next video.
18:36:59
Welcome back. Now, in the last video, we trained our first convolutional neural network. And from the looks of things, it's improved upon our baseline. But let's make sure by comparing, this is another important part of machine learning experiments is comparing the results across your experiments. So and training time. Now, we've done that in a way where we've got three dictionaries here of our model zero results, model one results, model two results. So how
18:37:28
about we create a data frame comparing them? So let's import pandas as PD. And we're going to compare results equals PD dot data frame. And because our model results dictionaries, uh, all have the same keys. Let's pass them in as a list. So model zero results, model one results, and model two results to compare them. Wonderful. And what it looks like when we compare the results. All righty. So recall our first model was our baseline V zero was just two linear layers.
18:38:09
And so we have an accuracy of 83.4 and a loss of 0.47. The next model was we trained on the GPU and we introduced nonlinearities. So we actually found that that was worse off than our baseline. But then we brought in the big guns. We brought in the tiny VGG architecture from the CNN explainer website and trained our first convolutional neural network. And we got the best results so far. But there's a lot more experiments that we could do. We could go back through our
18:38:43
tiny VGG and we could increase the number of hidden units. Where do we create our model up here? We could increase this to say 30 and see what happens. That would be a good experiment to try. And if we found that nonlinearities didn't help with our second model, we could comment out the relu layers. We could of course change the kernel size, change the padding, change the max pool. A whole bunch of different things that we could try here. We could train it for longer.
18:39:12
So maybe if we train it for 10 epochs, it would perform better. But these are just things to keep in mind and try out. I'd encourage you to give them a go yourself. But for now, we've kept all our experiments quite the same. How about we see the results we add in the training time? Because that's another important thing that we've been tracking as well. So we'll add training time to results comparison. So the reason why we do this is because
18:39:42
if this model is performing quite well, even compared to our CNN, so a difference in about 5% accuracy, maybe that's tolerable in the space that we're working, except that this model might actually train and perform inference 10 times faster than this model. So that's just something to be aware of. It's called the performance speed trade off. So let's add another column here, compare results. And we're going to add in, oh, excuse me, got a little error there. That's
18:40:12
all right. Got trigger happy on the shift and enter. Training time equals, we're going to add in, we've got another list here is going to be total train time for model zero, and total train time for model one, and total train time for model two. And then we have a look at our how compare results dictionary, or sorry, compare results data frame. Wonderful. So we see, and now this is another thing. I keep stressing this to keep in mind. If your numbers aren't exactly
18:40:50
of what I've got here, don't worry too much. Go back through the code and see if you've set up the random seeds correctly, you might need a koodle random seed. We may have missed one of those. If your numbers are out landishly different to these numbers, then you should go back through your code and see if there's something wrong. And again, the training time will be highly dependent on the compute environment you're using. So if you're running this notebook locally,
18:41:12
you might get faster training times. If you're running it on a different GPU to what I have, NVIDIA SMI, you might get different training times. So I'm using a Tesla P100, which is quite a fast GPU. But that's because I'm paying for Colab Pro, which generally gives you faster GPUs. And model zero was trained on the CPU. So depending on what compute resource Google allocates to you with Google Colab, this number might vary here. So just keep that in mind. These values training
18:41:43
time will be very dependent on the hardware you're using. But if your numbers are dramatically different, well, then you might want to change something in your code and see what's going on. And how about we finish this off with a graph? So let's go visualize our model results. And while we're doing this, have a look at the data frame above. Is the performance here 10 seconds longer training time worth that extra 5% of the results on the accuracy? Now in our case, we're using a
18:42:15
relatively toy problem. What I mean by toy problem is quite a simple data set to try and test this out. But in your practice, that may be worth doing. If your model takes longer to train, but gets quite a bit better performance, it really depends on the problem you're working with. Compare results. And we're going to set the index as the model name, because I think that's what we want our graph to be, not the model name. And then we're going to plot, we want to compare
18:42:43
the model accuracy. And we want to plot, the kind is going to be equal to bar h, horizontal bar chart. We've got p x label, we're going to get accuracy as a percentage. And then we're going to go py label. This is just something that you could share. If someone was asking, how did your modeling experiments go on fashion MNIST? Well, here's what I've got. And then they ask you, well, what's the fashion MNIST model V2? Well, you could say that's a convolutional neural network that
18:43:14
trained, that's replicates the CNN explainer website that trained on a GPU. How long did that take to train? Well, then you've got the training time here. We could just do it as a vertical bar chart. I did it as horizontal so that this looks a bit funny to me. So horizontal like that. So the model names are over here. Wonderful. So now I feel like we've got a trained model. How about we make some visual predictions? Because we've just got numbers on a page here,
18:43:45
but our model is trained on computer vision data. And the whole point of making a machine learning model on computer vision data is to be able to visualize predictions. So let's give that a shot, hey, in the next video, we're going to use our best performing model, fashion MNIST model V2 to make predictions on random samples from the test data set. You might want to give that a shot, make some predictions on random samples from the test data set, and plot them out with
18:44:13
their predictions as the title. So try that out. Otherwise, we'll do it together in the next video. In the last video, we compared our models results. We tried three experiments. One was a basic linear model. One was a linear model with nonlinear activations. And fashion MNIST model V2 is a convolutional neural network. And we saw that from an accuracy perspective, our convolutional neural network performed the best. However, it had the longest training time. And I just want to exemplify
18:44:45
the fact that the training time will vary depending on the hardware that you run on. We spoke about this in the last video. However, I took a break after finishing the last video, reran all of the cells that we've written, all of the code cells up here by coming back to the notebook and going run all. And as you'll see, if you compare the training times here to the last video, we get some different values. Now, I'm not sure exactly what hardware Google collab is using behind the
18:45:11
scenes. But this is just something to keep in mind, at least from now on, we know how to track our different variables, such as how long our model takes to train and what its performance values are. But it's time to get visual. So let's create another heading, make and evaluate. This is one of my favorite steps after training a machine learning model. So make and evaluate random predictions with the best model. So we're going to follow the data explorer's model of getting
18:45:44
visual visual visual or visualize visualize visualize. Let's make a function called make predictions. And it's going to take a model, which will be a torch and end module type. It's also going to take some data, which can be a list. It'll also take a device type, which will be torch dot device. And we'll set that by default to equal the default device that we've already set up. And so what we're going to do is create an empty list for prediction
18:46:12
probabilities. Because what we'd like to do is just take random samples from the test data set, make predictions on them using our model, and then plot those predictions. We want to visualize them. And so we'll also turn our model into evaluation mode, because if you're making predictions with your model, you should turn on evaluation mode. We'll also switch on the inference mode context manager, because predictions is another word for inference. And we're going to loop through
18:46:43
for each sample in data. Let's prepare the sample. So this is going to take in a single image. So we will unsqueeze it, because we need to add a batch size dimension on the sample, we'll set dim equals to zero, and then we'll pass that to the device. So add a batch dimension, that's with the unsqueeze, and pass to target device. That way, our data and model are on the same device. And we can do a forward pass. Well, we could actually up here go
18:47:21
model dot two device. That way we know that we've got device agnostic code there. Now let's do the forward pass, forward pass model outputs raw logits. So recall that if we have a linear layer at the end of our model, it outputs raw logits. So pred logit for a single sample is going to equal model. We pass the sample to our target model. And then we're going to get the prediction probability. How do we get the prediction probability? So we want to go from
18:47:50
logit to prediction probability. Well, if we're working with a multi class classification problem, we're going to use the softmax activation function on our pred logit. And we're going to squeeze it so it gets rid of an extra dimension. And we're going to pass in dim equals zero. So that's going to give us our prediction probability for a given sample. Now let's also turn our prediction probabilities into prediction labels. So get pred. Well, actually, I think we're just going
18:48:25
to return the pred probes. Yeah, let's see what that looks like, because we've got a an empty list up here for pred probes. So for matplotlib, we're going to have to use our data on the CPU. So let's make sure it's on the CPU, because matplotlib doesn't work with the GPU. So get pred prob off GPU for further calculations. So we're just hard coded in here to make sure that our prediction probabilities off the GPU. So pred probs, which is our list up here. We're
18:48:58
going to append the pred prob that we just calculated. But we're going to put it on the CPU. And then let's go down here. And we're going to. So if we've done it right, we're going to have a list of prediction probabilities relating to particular samples. So we're going to stack the pred probs to turn list into a tensor. So this is only one way of doing things. There are many different ways that you could make predictions and visualize them. I'm just exemplifying one way. So we're
18:49:30
going to torch stack, which is just going to say, hey, concatenate everything in the list to a single tensor. So we might need to tab that over, tab, tab. Beautiful. So let's try this function in action and see what happens. I'm going to import random. And then I'm going to set the random seed to 42. And then I'm going to create test samples as an empty list, because we want an empty or we want a list of test samples to iterate through. And I'm going to create test labels also as an
18:50:06
empty list. So that remember, when we are evaluating predictions, we want to compare them to the ground truth. So we want to get some test samples. And then we want to get their actual labels so that when our model makes predictions, we can compare them to their actual labels. So for sample, comma label, in, we're going to use random to sample the test data. Now note that this is not the test data loader. This is just test data. And we're going to set k equals to nine. And recall,
18:50:36
if you want to have a look at test data, what do we do here? We can just go test data,
18:50:41
which is our data set, not converted into a data loader yet. And then if we wanted to get the first 10 samples, can we do that? Only one element tensors can be converted into Python scalars. So if we get the first zero, and maybe we can go up to 10. Yeah, there we go. And what's the shape of this?
18:51:03
Tuple has no object shape. Okay, so we need to go image label equals that. And then can we check the shape of the image label? Oh, because the labels are going to be integers. Wonderful. So that's not the first 10 samples, but that's just what we get if we iterate through the test data, we get an image tensor, and we get an associated label. So that's what we're doing with this line here, we're just randomly sampling nine samples. And this could be any number you
18:51:37
want. I'm going to use nine, because this is a spoiler for later on, we're going to create a three by three plot. So that just nine is just a fun number. So get some random samples from the test data set. And then we can go test samples dot append sample. And we will go test labels dot append label. And then let's go down here, view the first, maybe we go first sample shape.
18:52:06
So test samples zero dot shape. And then if we get test samples, zero, we're going to get a tensor of image values. And then if we wanted to plot that, can we go PLT, M show, C map, equals gray. And we may have to squeeze this, I believe, to remove the batch tensor. Let's see what happens batch dimension. There we go. Beautiful. So that's to me, a shoe, a high heel shoe of some sort. If we get the title,
18:52:41
PLT dot title, test labels, let's see what this looks like. It's a five, which is, of course, class names will index on that. Sandal. Okay, beautiful. So we have nine random samples, nine labels that are associated with that sample. Now let's make some predictions. So make predictions. And this is one of my favorite things to do. I can't stress it enough is to randomly pick data samples from the test data set and predict on them and do it over and over and over again to see
18:53:18
what the model is doing. So not only at the start of a problem, I'll just get the prediction probabilities here. We're going to call our make predictions function. So not only at the start of a problem should you become one with the data, even after you've trained a model, you'll want to further become one with the data, but this time become one with your models predictions on the data and see what happens. So view the first two prediction probabilities list. So we're just
18:53:46
using our make predictions function that we created before, passing at the model, the train model to, and we're passing at the data, which is the test samples, which is this list that we just created up here, which is comprised of random samples from the test data set. Wonderful. So let's go. Pred probes. Oh, we don't want to view them all. That's going to give us
18:54:13
Oh, we want to the prediction probabilities for a given sample. And so how do we convert prediction probabilities into labels? Because if we're trying to, if we have a look at test labels, if we're trying to compare apples to apples, when we're evaluating our model, we want to, we can't really necessarily compare the prediction probabilities straight to the test labels. So we need to convert these prediction probabilities into prediction labels. So how can we do that?
18:54:44
Well, we can use argmax to take whichever value here, the index, in this case, this one, the index of whichever value is the highest of these prediction probabilities. So let's see that in action. Convert prediction probabilities to labels. So we'll go pred classes equals pred probes, and we'll get the argmax across the first dimension. And now let's have a look at the pred classes. Wonderful. So are they in the same format as our test labels? Yes, they are. So if
18:55:23
you'd like to go ahead, in the next video, we're going to plot these and compare them. So we're going to write some code to create a mapplotlib plotting function that's going to plot nine different samples, along with their original labels, and their predicted label. So give that a shot, we've just written some code here to make some predictions on random samples. If you'd like them to be truly random, you can comment out the seed here, but I've just kept the seed at 42. So that
18:55:51
our random dot sample selects the same samples on your end and on my end. So in the next video, let's plot these. Let's now continue following the data explorer's motto of visualize visualize visualize. We have some prediction classes. We have some labels we'd like to compare them to. You can compare them visually. It looks like our model is doing pretty good. But let's, since we're making predictions on images, let's plot those images along with the predictions.
18:56:22
So I'm going to write some code here to plot the predictions. I'm going to create a matplotlib figure. I'm going to set the fig size to nine and nine. Because we've got nine random samples, you could, of course, change this to however many you want. I just found that a three by three plot works pretty good in practice. And I'm going to set n rows. So for my matplotlib plot, I want three rows. And I want three columns. And so I'm going to enumerate through the samples in test
18:56:53
samples. And then I'm going to create a subplot for each sample. So create a subplot. Because this is going to create a subplot because it's within the loop. Each time it goes through a new sample, create a subplot of n rows and calls. And the index it's going to be on is going to be i plus one, because it can't start at zero. So we just put i plus one in there. What's going on here?
18:57:24
Enumerate. Oh, excuse me. In enumerate, wonderful. So now we're going to plot the target image. We can go plot dot in show, we're going to get sample dot squeeze. Because we need to remove the batch dimension. And then we're going to set the C map is equal to gray. What's this telling me up here? Oh, no, that's correct. Next, we're going to find the prediction label in text form, because we don't want it in a numeric form, we could do that. But we want to look at things
18:58:02
visually with human readable language, such as sandal for whatever class sandal is, whatever number class that is. So we're going to set the pred label equals class names. And we're going to index using the pred classes I value. So right now we're going to plot our sample. We're going to find its prediction. And now we're going to get the truth label. So we also want this in text form. And what is the truth label going to be? Well, the truth label is we're going to have to index
18:58:35
using class names and index on that using test labels I. So we're just matching up our indexes here. Finally, we're going to create a title, create a title for the plot. And now here's what I like to do as well. If we're getting visual, well, we might as well get really visual, right? So I think we can change the color of the title text, depending if the prediction is right or wrong. So I'm going to create a title using an F string, pred is going to be a pred label,
18:59:04
and truth label. We could even plot the prediction probabilities here if we wanted to. That might be an extension that you might want to try. And so here we're going to check for equality between pred and truth and change color of title text. So what I mean by this, it's going to be a lot easier to explain if we just if and doubt coded out. So if the pred label equals the truth label, so they're equal, I want the plot dot title to be the title text. But I want the font size,
18:59:41
well, the font size can be the same 10. I want the color to equal green. So if they're so green text, if prediction, same as truth, and else I'm going to set the plot title to have title text font size equals 10. And the color is going to be red. So does that make sense? All we're doing is we're enumerating through our test samples that we got up here, test samples that we found randomly from the test data set. And then each time we're creating a subplot, we're plotting our image,
19:00:19
we're finding the prediction label by indexing on the class names with our pred classes value,
19:00:24
we're getting the truth label, and we're creating a title for the plot that compares the pred label to the truth. And we're changing the color of the title text, depending if the pred label is correct or not. So let's see what happens. Did we get it right? Oh, yes, we did. Oh, I'm going to do one more thing. I want to turn off the accesses, just so we get more real estate. I love these kind of plots. It helps that our model got all of these predictions right. So look at this,
19:00:58
pred sandal, truth, sandal, pred trouser, truth trouser. So that's pretty darn good, right? See how, for me, I much appreciate, like, I much prefer visualizing things numbers on a page look good, but there's something, there's nothing quite like visualizing your machine learning models predictions, especially when it gets it right. So how about we select some different random samples up here, we could functionize this as well to do like all of this code in one hit, but that's all
19:01:24
right. We'll be a bit hacky for now. So this is just going to randomly sample with no seed at all. So your samples might be different to mine, nine different samples. So this time we have an ankle boot, we'll make some predictions, we'll just step through all of this code here. And oh, there we go. It got one wrong. So all of these are correct. But this is more interesting as well is where does your model get things wrong? So it predicted address, but this is a coat.
19:01:52
Now, do you think that this could be potentially address? To me, I could see that as being addressed. So I kind of understand where the model's coming from there. Let's make some more random predictions. We might do two more of these before we move on to the next video.
19:02:07
Oh, all correct. We're interested in getting some wrong here. So our model seems to be too good.
19:02:16
All correct again. Okay, one more time. If we don't get any wrong, we're going on to the next video. But this is just really, oh, there we go. Too wrong. Beautiful. So predicted address, and that's a shirt. Okay. I can kind of see where the model might have stuffed up there. It's a little bit long for a shirt for me, but I can still understand that that would be a shirt. And this is a pullover, but the truth is a coat. So maybe, maybe there's some issues with the labels.
19:02:42
And that's probably what you'll find in a lot of data sets, especially quite large ones. Just with a sheer law of large numbers, there may be some truth labels in your data sets that you work with that are wrong. And so that's why I like to see, compare the models predictions versus the truth on a bunch of random samples to go, you know what, is our models results better or worse than they actually are. And that's what visualizing helps you do is figure out,
19:03:07
you know what, our model is actually, it says it's good on the accuracy. But when we visualize the predictions, it's not too good. And vice versa, right? So you can keep playing around with this, try, look at some more random samples by running this again. We'll do one more for good luck. And then we'll move on to the next video. We're going to go on to another way. Oh, see, this is another example. Some labels here could be confusing. And speaking of confusing,
19:03:35
well, that's going to be a spoiler for the next video. But do you see how the prediction is a t-shirt top, but the truth is a shirt? To me, that label is kind of overlapping. Like, I don't know, what's the difference between a t-shirt and a shirt? So that's something that you'll find as you train models is maybe your model is going to tell you about your data as well.
19:03:58
And so we hinted that this is going to be confused. The model is confused between t-shirt top and shirt. How about we plot a confusion matrix in the next video? I'll see you there. We're up to a very exciting point in evaluating our machine learning model. And that is visualizing, visualizing, visualizing. And we saw that in the previous video, our model kind of gets a little bit confused. And in fact, I would personally get confused at the difference
19:04:30
between t-shirt slash top and a shirt. So these kind of insights into our model predictions can also give us insights into maybe some of our labels could be improved. And another way to check that is to make a confusion matrix. So let's do that, making a confusion matrix for further prediction evaluation. Now, a confusion matrix is another one of my favorite ways of evaluating a classification model, because that's what we're doing. We're doing multi class classification.
19:05:04
And if you recall, if we go back to section two of the lone pytorch.io book, and then if we scroll down, we have a section here, more classification evaluation metrics. So accuracy is probably the gold standard of classification evaluation. There's precision, there's recall, there's F1 score, and there's a confusion matrix here. So how about we try to build one of those? I want to get this and copy this.
19:05:30
So, and write down a confusion matrix is a fantastic way of evaluating your classification models
19:05:43
visually. Beautiful. So we're going to break this down. First of all, we need to plot a confusion matrix. We need to make predictions with our trained model on the test data set. Number two, we're going to make a confusion matrix. And to do so, we're going to leverage torch metrics tricks have to figure out how to spell metrics and confusion matrix. So recall that torch metrics we've touched on this before is a great package torch metrics for a whole
19:06:18
bunch of evaluation metrics of machine learning models in pytorch flavor. So if we find we've got classification metrics, we've got audio image detection. Look how this is beautiful, a bunch of different evaluation metrics. And if we go down over here, we've got confusion matrix. So I only touched on five here, but or six. But if you look at torch metrics, they've got, how many is that about 25 different classification metrics? So if you want some extra curriculum,
19:06:48
you can read through these. But let's go to confusion matrix. And if we look at some code here, we've got torch metrics, confusion matrix, we need to pass in number of classes. We can normalize if we want. And do you notice how this is quite similar to the pytorch documentation? Well, that's the beautiful thing about torch metrics is that it's created with pytorch in mind. So let's try out if you wanted to try it out on some
19:07:16
tester code, you could do it here. But since we've already got some of our own code, let's just bring in this. And then number three is to plot it. We've got another helper package here, plot the confusion matrix using ML extend. So this is another one of my favorite helper libraries for machine learning things. It's got a lot of functionality that you can code up yourself, but you often find yourself coding at a few too many times, such as plotting a confusion
19:07:47
matrix. So if we look up ML extend plot confusion matrix, this is a wonderful library. I believe it was
19:07:58
it was created by Sebastian Rushka, who's a machine learning researcher and also author of a great book. There he is. Yeah, this is a side note machine learning with pytorch and scikit loan. I just got this book it just got released in the start of 2022. And it's a great book. So that's a little side note for learning more about machine learning with pytorch and scikit loan. So shout out to Sebastian Rushka. Thank you for this package as well. This is going to
19:08:27
just help us plot a confusion matrix like this. So we'll have our predicted labels on the bottom and our true labels on the side here. But we can just copy this code in here. Link sorry, and then confusion matrix, we can copy that in here. The thing is that torch metrics doesn't come with Google Colab. So if you're using Google Colab, I think ML extend does, but we need a certain version of ML extend that Google Colab doesn't yet have yet. So we actually
19:08:58
need version 0.19.0. But we're going to import those in a second. Let's first make some predictions across our entire test data set. So previously, we made some predictions only on nine random samples.
19:09:13
So random sample, we selected nine. You could, of course, change this number to make it on more. But this was only on nine samples. Let's write some code to make predictions across our entire test data set. So import tqdm.auto for progress bar tracking. So tqdm.auto. We don't need to re-import it. I believe we've already got it above, but I'm just going to do it anyway for completeness. And so we're going to make, this is step one, above,
19:09:45
make predictions, make predictions with trained model. Our trained model is model two. So let's create an empty predictions list. So we can add our predictions to that. We're going to set our model into evaluation mode. And we're going to set with torch inference mode as our context manager. And then inside that, let's just build the same sort of code that we used for our testing loop,
19:10:12
except this time we're going to append all of our predictions to a list.
19:10:18
So we're going to iterate through the test data loader. And we can give our tqdm description. We're going to say making predictions dot dot dot. You'll see what that looks like in a minute.
19:10:31
And here we are going to send the data and targets to target device. So x, y equals x
19:10:42
to device and y to device. Wonderful. And we're going to do the forward pass. So we're going to create y logit. Remember, the raw outputs of a model with a linear layer at the end are referred to as logits. And we don't need to calculate the loss, but we want to turn predictions
19:11:03
from logits to prediction probabilities to prediction labels. So we'll set here y pred equals torch dot softmax. You could actually skip the torch softmax step if you wanted to and just take the argmax of the logits. But we will just go from prediction probabilities to pred labels for completeness. So squeeze and we're going to do it across the first dimension or the zeroth dimension. And then we'll take the argmax of that across the first dimension as well. And a little tidbit. If you
19:11:40
take different dimensions here, you'll probably get different values. So just check the inputs and outputs of your code to make sure you're using the right dimension here. And so let's go put predictions on CPU for evaluation. Because if we're going to plot anything, that plot lib will want them on the CPU. So we're going to append our predictions to y preds, y pred dot CPU. Beautiful. And because we're going to have a list of different predictions, we can use concatenate
19:12:10
a list of predictions into a tensor. So let's just print out y preds. And so I can show you what it looks like. And then if we go y pred tensor, this is going to turn our list of predictions into a single tensor. And then we'll go y pred tensor. And we'll view the first 10. Let's see if this works. So making predictions. Oh, would you look at that? Okay, so yeah, here's our list of predictions. A big list of tensors. Right, we don't really want it like that. So if we get rid of
19:12:46
that, and there's our progress bar, it's going through each batch in the test data load, so there's 313 batches of 32. So if we comment out print y preds, this line here torch dot cat y preds is going to turn this these tensors into a single tensor, or this list of tensors into a single tensor concatenate. Now, if we have a look, there we go, beautiful. And if we have a look at the whole thing, we're making predictions every single time here, but that's all right. They are pretty
19:13:17
quick. There we go. One big long tensor. And then if we check length y pred tensor, there should be one prediction per test sample. 10,000 beautiful. So now we're going to, we need to install torch metrics because torch metrics doesn't come with Google Colab at the time of recording. So let me just show you if we tried to import torch metrics. It doesn't, it might in the future, so just keep that in mind, it might come with Google Colab because it's a pretty useful package. But let's
19:13:50
now install see if required packages are installed. And if not, install them. So we'll just install torch metrics. We'll finish off this video by trying to import. We'll set up a try and accept loop. So Python is going to try import torch metrics and ML extend. I write it like this, because you may already have to which metrics and ML extend if you're running this code on a local machine. But if you're running it in Google Colab, which I'm sure many of you are, we are
19:14:23
going to try and import it anyway. And if it doesn't work, we're going to install it. So ML extend, I'm just going to check the version here because we need version for our plot confusion matrix function. This one, we need version 0.19.0 or higher. So I'm just going to write a little statement here. Assert int ML extend dot version. So if these two, if this condition in the try loop is or try block is accepted, it will skip the next step dot split. And I'm just going to check
19:15:02
the first index string equals is greater than or equal to 19. Otherwise, I'm going to return an error saying ML extend version should be 0.19.0 or higher. And so let me just show you what this looks like. If we run this here, string and int, did I not turn it into a string? Oh, excuse me. There we go. And I don't need that bracket on the end. There we go. So that's what I'm saying. So this is just saying, hey, the version of ML extend that you have should be 0 or should be
19:15:47
19 or higher. Because right now Google Colab by default has 14, this may change in the future. So let's finish off this accept block. If the above condition fails, which it should, we are going to pip install. So we're going to install this into Google Colab torch metrics. We're going to do it quietly. And we're also going to pass the U tag for update ML extend. So import torch metrics, ML extend afterwards, after it's been installed and upgraded. And print,
19:16:20
we're going to go ML extend version, going to go ML extend underscore version. And let's see what happens if we run this. So we should see, yeah, some installation happening here. This is going to install torch metrics. Oh, do we not have ML extend the upgraded version? Let's have a look. We may need to restart our Google Colab instance. Ah, okay, let's take this off. Quiet.
19:16:55
Is this going to tell us to restart Google Colab?
19:17:00
Well, let's restart our runtime. After you've run this cell, if you're using Google Colab, you may have to restart your runtime to reflect the fact that we have the updated version of ML extend. So I'm going to restart my runtime now. Otherwise, we won't be able to plot our confusion matrix. We need 0.19.0. And I'm going to run all of these cells. So I'm going to pause the video here, run all of the cells by clicking run all. Note, if you run into any errors, you will have
19:17:31
to run those cells manually. And then I'm going to get back down to this cell and make sure that I have ML extend version 0.1.9. I'll see in a few seconds.
19:17:46
I'm back. And just a little heads up. If you restart your runtime and click run all, your Colab notebook will stop running cells if it runs into an error. So this is that error we found in a previous video where our data and model were on different devices. So to skip past that, we can just jump to the next cell and we can click run after. There we go. And it's going to run all of the cells after for us. It's going to retrain our models. Everything's going to get rerun.
19:18:15
And then we're going to come right back down to where we were before trying to install the updated version of ML extend. I'm going to write some more code while our code is running import ML extend. And then I'm going to just make sure that we've got the right version here. You may require a runtime restart. You may not. So just try to see after you've run this install of torch metrics and upgrade of ML extend. See if you can re import ML extend. And if you have the
19:18:43
version 0.19.0 or above, we should be able to run the code. Yeah, there we go. Wonderful. ML extend 0.19.0. And we've got ML extend version, assert, import. Beautiful. So we've got a lot of extra code here. In the next video, let's move forward with creating a confusion matrix. I just wanted to show you how to install and upgrade some packages in Google Colab if you don't have them. But now we've got predictions across our entire test data set. And we're going
19:19:18
to be moving towards using confusion matrix function here to compare our predictions versus the target data of our test data set. So I'll see in the next video, let's plot a confusion matrix.
19:19:36
Welcome back. In the last video, we wrote a bunch of code to import some extra libraries that we need for plotting a confusion matrix. This is really helpful, by the way. Google Colab comes with a lot of prebuilt installed stuff. But definitely later on down the track, you're going to need to have some experience installing stuff. And this is just one way that you can do it. And we also made predictions across our entire test data set. So we've got 10,000 predictions in this tensor.
19:20:01
And what we're going to do with a confusion matrix is confirm or compare these predictions to the target labels in our test data set. So we've done step number one. And we've prepared ourselves for step two and three, by installing torch metrics, and installing ML extend or the later version of ML extend. So now let's go through step two, making a confusion matrix, and step three plotting that confusion matrix. This is going to look so good. I love how good
19:20:30
confusion matrix is look. So because we've got torch metrics now, we're going to import the confusion matrix class. And from our ML extend, we're going to go into the plotting module, and import plot confusion matrix. Recall that the documentation for both of these are within torch metrics here, and within ML extend here. Let's see what they look like. So number two is set up confusion matrix instance, and compare predictions to targets. That's what evaluating a
19:21:07
model is, right? Comparing our models predictions to the target predictions. So I'm going to set up a confusion matrix under the variable conf mat, then I'm going to call the confusion matrix class from torch metrics. And to set up an instance of it, I need to pass in the number of classes that we have. So because we have 10 classes, they are all contained within class names. Recall that class names is a list of all of the different classes that we're working with. So I'm just going
19:21:36
to pass in the number of classes as the length of our class names. And then I can use that conf mat instance, confusion matrix instance, to create a confusion matrix tensor by passing into conf mat, which is what I've just created up here. Conf mat, just like we do with our loss function, I'm going to pass in preds equals our Y pred tensor, which is just above Y pred tensor that we calculated all of the predictions on the test data set. There we go. That's our preds.
19:22:10
And our target is going to be equal to test data dot targets. And this is our test data data set that we've seen before. So if we go test data and press tab, we've got a bunch of different attributes, we can get the classes. And of course, we can get the targets, which is the labels. PyTorch calls labels targets. I usually refer to them as labels, but the target is the test data target. So we want to compare our models predictions on the test data set to our test data targets.
19:22:44
And so let's keep going forward. We're up to step number three now. So this is going to create our confusion matrix tensor. Oh, let's see what that looks like, actually. Conf mat tensor.
19:22:58
Oh, okay. So we've got a fair bit going on here. But let's turn this into a pretty version of this. So along the bottom is going to be our predicted labels. And along the side here is going to be our true labels. But this is where the power of ML extend comes in. We're going to plot our confusion matrix. So let's create a figure and an axes. We're going to call the function plot confusion matrix that we've just imported above. And we're going to pass in our conf mat equals
19:23:32
our conf mat tensor. But because we're working with map plot lib, it'll want it as NumPy.
19:23:39
So I'm just going to write here, map plot lib likes working with NumPy. And we're going to pass in the class names so that we get labels for each of our rows and columns. Class names, this is just a list of our text based class names. And then I'm going to set the fig size to my favorite hand and poker, which is 10, seven. Also happens to be a good dimension for Google Colab. Look at that. Oh, that is something beautiful to see. Now a confusion matrix. The
19:24:12
ideal confusion matrix will have all of the diagonal rows darkened with all of the values and no values here, no values here. Because that means that the predicted label lines up with the true label. So in our case, we have definitely a very dark diagonal here. But let's dive into some of the highest numbers here. It looks like our model is predicting shirt when the true label is actually t shirt slash top. So that is reflective of what we saw before. Do we still have that
19:24:40
image there? Okay, we don't have an image there. But in a previous video, we saw that when we plotted our predictions, the model predicted t shirt slash top when it was actually a shirt. And of course, vice versa. So what's another one here? Looks like our model is predicting shirt when it's actually a coat. And now this is something that you can use to visually inspect your data to see if the the errors that your model is making make sense from a visual perspective. So it's getting
19:25:12
confused by predicting pull over when the actual label is coat, predicting pull over when the actual label is shirt. So a lot of these things clothing wise and data wise may in fact look quite the same. Here's a relatively large one as well. It's predicting sneaker when it should be an ankle boot. So it's confusing two different types of shoes there. So this is just a way to further evaluate your model and start to go. Hmm, maybe our labels are a little bit confusing.
19:25:38
Could we expand them a little bit more? So keep that in mind, a confusion matrix is one of the most powerful ways to visualize your classification model predictions. And a really, really, really helpful way of creating one is to use torch metrics confusion matrix. And to plot it, you can use plot confusion matrix from ML extend. However, if you're using Google Colab for these, you may need to import them or install them. So that's a confusion matrix. If you'd like
19:26:08
more classification metrics, you've got them here. And you've got, of course, more in torch metrics. So give that a look. I think in the next video, we've done a fair bit of evaluation. Where are we up to in our workflow? I believe it's time we saved and loaded our best trained model. So let's give that a go. I'll see you in the next video.
19:26:31
In the last video, we created a beautiful confusion matrix with the power of torch metrics and ML extend. But now it's time to save and load our best model. Because if we, if we evaluated it, our convolutional neural network and go, you know what, this model is pretty good. Let's export it to a file so we can use it somewhere else. Let's see how we do that. And by the way, if we go into our keynote, we've got a value at model torch metrics. We've been through this a fair few times
19:27:02
now. We've improved through experimentation. We haven't used tensor board yet, but that'll be in a later video and save and reload your trained model. So here's where we're up to. If we've gone through all these steps enough times and we're like, you know what, let's save our model so we can use it elsewhere. And we can reload it in to make sure that it's, it's saved correctly. Let's go through with this step. We want number 11. We're going to go save and load
19:27:26
best performing model. You may have already done this before. So if you've been through the other parts of the course, you definitely have. So if you want to give that a go, pause the video now and try it out yourself. I believe we did it in notebook number one. We have here we go, saving and loading a pie torch model. You can go through this section of section number one on your own and see if you can do it. Otherwise, let's code it out together. So I'm going to start
19:27:53
from with importing path from path lib, because I like to create a model directory path. So create model directory path. So my model path is going to be set equal to path. And I'm going to save it to models. This is where I want to, I want to create file over here called models and save my models to their model path dot MKD for make directory parents. Yes, I wanted to make the parent directories if they don't exist and exist. Okay. Also equals true. So if we try to
19:28:28
create it, but it's already existing, we're not going to get an error. That's fine. And next, we're going to create a model save path. Just going to add some code cells here. So we have more space. Let's pass in here a model name. Going to set this equal to, since we're on section three, I'm going to call this O three pie torch, computer vision, model two is our best model. And I'm going to save it to PTH for pie torch. You can also save it to dot PT. I like to use PTH. And we're
19:29:04
going to go model save path equal model path slash model name. So now if we have a look at this, we're going to have a path called model save path. But it's going to be a POSIX path in models O three pie torch computer vision, model two dot PTH. And if we have a look over here, we should have, yeah, we have a models directory now. That's not going to have anything in it at the moment. We've got our data directory that we had before there's fashion MNIST. This is a good
19:29:34
way to start setting up your directories, break them down data models, helper function files, etc. But let's keep going. Let's save, save the model state dict. We're going to go print,
19:29:49
saving model to just going to give us some information about what's happening. Model save path. And we can save a model by calling torch dot save. And we pass in the object that we want to save using the object parameter, OBJ. When we get a doc string there, we're going to go model two, we want to save the state dict, recall that the state dict is going to be our models what our models learned parameters on the data set, so that all the weights and biases and all that
19:30:21
sort of jazz. Beautiful. So when we first created model two, these were all random numbers. They've been or since we trained model two on our training data, these have all been updated to represent the training images. And we can leverage these later on, as you've seen before, to make predictions. So I'm not going to go through all those, but that's what we're saving. And the file path is going to be our model save path. So let's run this and see what happens. Beautiful. We're saving our
19:30:52
model to our model directory. And now let's have a look in here. Do we have a model? Yes, we do. Beautiful. So that's how quickly we can save a model. Of course, you can customize what the name is, where you save it, et cetera, et cetera. Now, let's see what happens when we load it in. So create a new instance, because we only saved the state dict of model two, we need to create a new instance of our model two, or how it was created, which was with
19:31:20
our class fashion MNIST V two. If we saved the whole model, we could just import it to a new variable. But I'll let you read back more on that on the different ways of saving a model in here. There's also a link to the pytorch documentation would highly recommend that. But let's see it in action, we need to create a new instance of our fashion MNIST model V two, which is our convolution or neural network. So I'm going to set the manual seed. That way when we create a new instance,
19:31:50
it's instantiated with the same random numbers. So we're going to set up loaded model two, equals fashion MNIST V two. And it's important here that we set it up with the same parameters as our original saved model. So fashion MNIST V two. Oh, we've got a typo here.
19:32:11
I'll fashion MNIST model V two. Wonderful. So the input shape is going to be one, because that is the number of color channels in our test, in our images, test image dot shape. Do we still have a test image should be? Oh, well, we've created a different one, but our image size, our image shape is 12828 image shape for color channels height width. Then we create it with hidden units, we use 10 for hidden units. So we can just set that here. This is important,
19:32:43
they just have to otherwise if the shapes aren't the same, what are we going to get? We're going to get a shape mismatch error. And our output shape is what is also going to be 10 or length of class names. If you have the class names variable instantiated, that is. So we're going to load in the saved state dict, the one that we just saved. So we can go loaded model two,
19:33:07
dot load state dict. And we can pass in torch dot load in here. And the file that we want to load or the file path is model save path up here. This is why I like to just save my path variables to a variable so that I can just use them later on, instead of re typing out this all the time, which is definitely prone to errors. So we're going to send the model to the target device. Loaded model two dot two device. Beautiful. Let's see what happens here.
19:33:46
Wonderful. So let's now evaluate the loaded model. So evaluate loaded model. The results should be very much the same as our model two results. So model two results. So this is what we're looking for. We want to make sure that our saved model saved these results pretty closely. Now I say pretty closely because you might find some discrepancies in this lower these lower decimals here, just because of the way files get saved and something gets lost,
19:34:18
et cetera, et cetera. So that's just to do with precision and computing. But as long as the first few numbers are quite similar, well, then we're all gravy. So let's go torch manual seed.
19:34:33
Remember, evaluating a model is almost as well is just as important as training a model. So this is what we're doing. We're making sure our model save correctly. Before we deployed it, if it didn't if we deployed it, it didn't save correctly. Well, then we'd get our we would get less than ideal results, wouldn't we? So model equals loaded model two, we're going to use our same of our model function, by the way. And of course, we're going to evaluate it on the same test data
19:35:00
set that we've been using test data loader. And we're going to create a loss function or just put in our loss function that we've created before. And our accuracy function is the accuracy function we've been using throughout this notebook. So now let's check out loaded model two results. They should be quite similar to this one. We're going to make some predictions. And then if we go down, do we have the same numbers? Yes, we do. So we have five, six, eight, two, nine, five, six,
19:35:28
eight, two, nine, wonderful. And three, one, three, five, eight, three, one, three, five, eight, beautiful. It looks like our loaded model gets the same results as our previously trained model before we even saved it. And if you wanted to check if they were close, you can also use torch dot is close, check if model results, if you wanted to check if they were close programmatically, that is, because we just looked at these visually, check if model results are close to each other.
19:35:58
Now we can go torch is close, we're going to pass in torch dot tensor, we have to turn these values into a tensor. We're going to go model two results. And we'll compare the model loss. How about we do that? We want to make sure the loss values are the same. Or very close, that is with torch dot is close. Torch dot tensor model. Or we want this one to be loaded model two results. Model loss. Another bracket on the end there. And we'll see how close they are true,
19:36:35
wonderful. Now, if this doesn't return true, you can also adjust the tolerance levels in here. So we go atal equals, this is going to be the absolute tolerance. So if we do one to the negative eight, it's saying like, Hey, we need to make sure our results are basically the same up to eight decimal points. That's probably quite low. I would say just make sure they're at least within two. But if you're getting discrepancies here between your saved model and your loaded model, or sorry,
19:37:05
this model here, the original one and your loaded model, if they are quite large, so they're like more than a few decimal points off in this column or even here, I'd go back through your code and make sure that your model is saving correctly, make sure you've got random seeds set up. But if they're pretty close, like in terms of within three or two decimal places of each other, well, then I'd say that's that's close enough. But you can also adjust the tolerance level here
19:37:29
to check if your model results are close enough, programmatically. Wow, we have covered a fair bit here. We've gone through this entire workflow for a computer vision problem. Let's in the next video, I think that's enough code for this section, section three, pytorch computer vision. I've got some exercises and some extra curriculum lined up for you. So let's have a look at those in the next video. I'll see you there. My goodness. Look how much computer vision pytorch code
19:38:03
we've written together. We started off right up the top. We looked at the reference notebook and the online book. We checked out computer vision libraries and pytorch, the main one being torch vision. Then we got a data set, namely the fashion MNIST data set. There are a bunch more data sets that we could have looked at. And in fact, I'd encourage you to try some out in the torch vision dot data sets, use all of the steps that we've done here to try it on another data set. We repaired
19:38:28
our data loaders. So turned our data into batches. We built a baseline model, which is an important step in machine learning, because the baseline model is usually relatively simple. And it's going to serve as a baseline that you're going to try and improve upon through just go back to the keynote through various experiments. We then made predictions with model zero. We evaluated it. We timed our predictions to see if running our models on the GPU was faster when we learned that
19:38:57
sometimes a GPU won't necessarily speed up code if it's a relatively small data set because of the overheads between copying data from CPU to GPU. We tried a model with non-linearity and we saw that it didn't really improve upon our baseline model. But then we brought in the big guns, a convolutional neural network, replicating the CNN explainer website. And by gosh, didn't we spend a lot of time here? I'd encourage you as part of your extra curriculum to go through this again and again.
19:39:27
I still even come back to refer to it too. I referred to it a lot making the materials for this video section and this code section. So be sure to go back and check out the CNN explainer website for more of what's going on behind the scenes of your CNNs. But we coded one using pure pytorch. That is amazing. We compared our model results across different experiments. We found that our convolutional neural network did the best, although it took a little bit longer to train. And we also
19:39:55
learned that the training time values will definitely vary depending on the hardware you're using. So that's just something to keep in mind. We made an evaluated random predictions with our best model, which is an important step in visualizing, visualizing, visualizing your model's predictions, because you could get evaluation metrics. But until you start to actually visualize what's going on, well, in my case, that's how I best understand what my model is thinking. We saw a confusion
19:40:24
matrix using two different libraries torch metrics and ML extend a great way to evaluate your classification models. And we saw how to save and load the best performing model to file and made sure that the results of our saved model weren't too different from the model that we trained within the notebook. So now it is time I'd love for you to practice what you've gone through. This is actually really exciting now because you've gone through an end-to-end
19:40:52
computer vision problem. I've got some exercises prepared. If you go to the learn pytorch.io website in section 03, scroll down. You can read through all of this. This is all the materials that we've just covered in pure code. There's a lot of pictures in this notebook too that are helpful to learn things what's going on. We have some exercises here. So all of the exercises are focused on practicing the code and the sections above. We have two resources. We also have some extra
19:41:19
curriculum that I've put together. If you want an in-depth understanding of what's going on behind the scenes in the convolutional neural networks, because we've focused a lot on code, I'd highly recommend MIT's induction to deep computer vision lecture. You can spend 10 minutes clicking through the different options in the pytorch vision library, torch vision, look up most common convolutional neural networks in the torch vision model library, and then for a larger number
19:41:44
of pre-trained pytorch computer vision models, and if you get deeper into computer vision, you're probably going to run into the torch image models library, otherwise known as 10, but I'm going to leave that as extra curriculum. I'm going to just link this exercises section here. Again, it's at learn pytorch.io in the exercises section. We come down. There we go.
19:42:07
But there is also resource here, an exercise template notebook. So we've got one, what are three areas in industry where computer vision is being currently used. Now this is in the pytorch deep learning repo, extras exercises number three. I've put out some template code here for you to fill in these different sections. So some of them are code related. Some of them are just text based, but they should all be able to be completed by referencing what we've gone
19:42:35
through in this notebook here. And just as one more, if we go back to pytorch deep learning,
19:42:43
this will probably be updated by the time you get here, you can always find the exercise in extra curriculum by going computer vision, go to exercise in extra curriculum, or if we go into the extras file, and then we go to solutions. I've now also started to add video walkthroughs of each of the solutions. So this is me going through each of the exercises myself and coding them. And so you'll get to see the unedited videos. So they're just one long live stream.
19:43:12
And I've done some for O2, O3, and O4, and there will be more here by the time you watch this video. But if you'd like to see how I figure out the solutions to the exercises, you can watch those videos and go through them yourself. But first and foremost, I would highly recommend trying out the exercises on your own first. And then if you get stuck, refer to the notebook here, refer to the pytorch documentation. And finally, you can check out what I would have coded as a
19:43:40
potential solution. So there's number three, computer vision, exercise solutions. So congratulations on going through the pytorch computer vision section. I'll see you in the next section. We're going to look at pytorch custom data sets, but no spoilers. I'll see you soon.
19:44:04
Hello, hello, hello, and welcome to section number four of the Learn pytorch for deep learning course. We have custom data sets with pytorch. Now, before we dive into what we're going to cover, let's answer the most important question. Where can you get help? Now, we've been through this a few times now, but it's important to reiterate. Follow along with the code as best you can. We're going to be writing a bunch of pytorch code. Remember the motto, if and out, run the code.
19:44:37
That's in line with try it for yourself. If you'd like to read or read the doxtring, you can press shift command plus space in Google Colab. Or if you're on Windows, command might be control. Then if you're still stuck, you can search for it. Two of the resources you will probably come across is stack overflow or the wonderful pytorch documentation, which we've had a lot of experience with so far. Then, of course, try again, go back through your code,
19:45:03
if and out, code it out, or if and out, run the code. And then finally, if you're still stuck, ask a question on the pytorch deep learning discussions GitHub page. So if I click this link, we come to Mr. D Burke slash pytorch deep learning, the URL is here. We've seen this before. If you have a trouble or a problem with any of the course, you can start a discussion and you can select the category, general ideas, polls, Q and A, and then we can go here, video,
19:45:34
put the video number in. So 99, for example, my code doesn't do what I'd like it to. So say your problem and then come in here, write some code here, code here, and then my question is
19:45:54
something, something, something, click start discussion, and then we can help out. And then if we come back to the discussions, of course, you can search for what's going on. So if you have an error and you feel like someone else might have seen this error, you can, of course, search it and find out what's happening. Now, I just want to highlight again, the resources for this course are at learn pytorch.io. We are up to section four. This is a beautiful online book version of
19:46:17
all the materials we are going to cover in this section. So spoiler alert, you can use this as a reference. And then, of course, in the GitHub, we have the same notebook here, pytorch custom data sets. This is the ground truth notebook. So check that out if you get stuck. So I'm just going to exit out of this. We've got pytorch custom data sets at learn pytorch.io. And then, of course, the discussions tab for the Q&A. Now, if we jump back to the keynote, what do we have?
19:46:46
We might be asking, what is a custom data set? Now, we've built a fair few pytorch deeplining neural networks so far on various data sets, such as fashion MNIST. But you might be wondering, hey, I've got my own data set, or I'm working on my own problem. Can I build a model with pytorch to predict on that data set? And the answer is yes. However, you do have to go through a few pre processing steps to make that data set compatible with pytorch. And that's what we're
19:47:17
going to be covering in this section. And so I'd like to highlight the pytorch domain libraries. Now, we've had a little bit of experience before with torch vision, such as if we wanted to classify whether a photo was a pizza, steak, or sushi. So a computer vision image classification problem. Now, there's also text, such as if these reviews are positive or negative. And you can use torch text for that. But again, these are only just one problem within the vision space within the text
19:47:48
space. I want you to just understand that if you have any type of vision data, you probably want to look into torch vision. And if you have any kind of text data, you probably want to look into torch text. And then if you have audio, such as if you wanted to classify what song was playing, this is what Shazam does, it uses the input sound of some sort of music, and then runs a neural network over it to classify it to a certain song, you can look into torch audio for that. And then if you'd
19:48:18
like to recommend something such as you have an online store, or if your Netflix or something like that, and you'd like to have a homepage that updates for recommendations, you'd like to look into torch rec, which stands for recommendation system. And so this is just something to keep in mind.
19:48:36
Because each of these domain libraries has a data sets module that helps you work with different data sets from different domains. And so different domain libraries contain data loading functions for different data sources. So torch vision, let's just go into the next slide, we have problem space vision for pre built data sets, so existing data sets like we've seen with fashion MNIST, as well as functions to load your own vision data sets, you want to look into torch vision
19:49:07
dot data sets. So if we click on this, we have built in data sets, this is the pie torch documentation. And if we go here, we have torch audio, torch text, torch vision, torch rec, torch data. Now, at the time of recording, which is April 2022, this is torch data is currently in beta. But it's going to be updated over time. So just keep this in mind, updated over time to add even more ways to load different data resources. But for now, we're just going to get familiar with torch vision
19:49:38
data sets. If we went into torch text, there's another torch text dot data sets. And then if we went into torch audio, we have torch audio dot data sets. And so you're noticing a trend here that depending on the domain you're working in, whether it be vision, text, audio, or your data is recommendation data, you'll probably want to look into its custom library within pie torch. And of course, the bonus is torch data. It contains many different helper functions for loading data,
19:50:09
and is currently in beta as of April 2022. So 2022. So the by the time you watch this torch data may be out of beta. And then that should be something that's extra curriculum on top of what we're going to cover in this section. So let's keep going. So this is what we're going to work towards building food vision mini. So we're going to load some data, namely some images of pizza, sushi, and steak from the food 101 data set, we're going to build an image classification model,
19:50:42
such as the model that might power a food vision recognition app or a food image recognition app. And then we're going to see if it can classify an image of pizza as pizza, an image of sushi as sushi, and an image of steak as steak. So this is what we're going to focus on. We want to load, say we had images existing already of pizza, sushi, and steak, we want to write some code to load these images of food. So our own custom data set for building this food vision mini model,
19:51:13
which is quite similar to if you go to this is the project I'm working on personally, neutrify.app. This is a food image recognition model. Here we go. So it's still a work in progress as I'm going through it, but you can upload an image of food and neutrify will try to classify what type of food it is. So do we have steak? There we go. Let's upload that. Beautiful steak. So we're going to be building a similar model to what powers neutrify. And then there's the
19:51:45
macro nutrients for the steak. If you'd like to find out how it works, I've got all the links here, but that's at neutrify.app. So let's keep pushing forward. We'll go back to the keynote. This is what we're working towards. As I said, we want to load these images into PyTorch so that we can build a model. We've already built a computer vision model. So we want to figure out how do we get our own data into that computer vision model. And so of course we'll be adhering
19:52:13
to our PyTorch workflow that we've used a few times now. So we're going to learn how to load a data set with our own custom data rather than an existing data set within PyTorch. We'll see how we can build a model to fit our own custom data set. We'll go through all the steps that's involved in training a model such as picking a loss function and an optimizer. We'll build a training loop. We'll evaluate our model. We'll improve through experimentation. And then we can see save and reloading
19:52:44
our model. But we're also going to practice predicting on our own custom data, which is a very, very important step whenever training your own models. So what we're going to cover broadly,
19:52:57
we're going to get a custom data set with PyTorch. As we said, we're going to become one with the data. In other words, preparing and visualizing it. We'll learn how to transform data for use with a model, very important step. We'll see how we can load custom data with pre-built functions and our own custom functions. We'll build a computer vision model, aka food vision mini, to classify pizza, steak, and sushi images. So a multi-class classification model. We'll compare
19:53:24
models with and without data augmentation. We haven't covered that yet, but we will later on. And finally, we'll see how we can, as I said, make predictions on custom data. So this means data that's not within our training or our test data set. And how are we going to do it? Well, we could do it cooks or chemists. But I like to treat machine learning as a little bit of an art, so we're going to be cooking up lots of code. With that being said, I'll see you in Google Colab.
19:53:54
Let's code. Welcome back to the PyTorch cooking show. Let's now learn how we can cook up some custom data sets. I'm going to jump into Google Colab. So colab.research.google.com.
19:54:12
And I'm going to click new notebook. I'm just going to make sure this is zoomed in enough for the video. Wonderful. So I'm going to rename this notebook 04 because we're up to section 04. And I'm going to call it PyTorch custom data sets underscore video because this is going to be one of the video notebooks, which has all the code that I write during the videos, which is of course contained within the video notebooks folder on the PyTorch deep learning repo. So if you'd like
19:54:44
the resource or the ground truth notebook for this, I'm going to just put a heading here.
19:54:49
04 PyTorch custom data sets video notebook, make that bigger, and then put resources.
19:55:01
So book version of the course materials for 04. We'll go there, and then we'll go ground truth version of notebook 04, which will be the reference notebook that we're going to use for this section. Come into PyTorch custom data sets. And then we can put that in there. Wonderful. So the whole synopsis of this custom data sets section is we've used some data sets with PyTorch before, but how do you get your own data into PyTorch? Because that's what you
19:55:44
want to start working on, right? You want to start working on problems of your own. You want to come into any sort of data that you've never worked with before, and you want to figure out how do you get that into PyTorch. So one of the ways to do so is via custom data sets. And then I want to put a note down here. So we're going to go zero section zero is going to be importing PyTorch and setting up device agnostic code. But I want to just stress here that domain libraries.
19:56:23
So just to reiterate what we went through last video. So depending on what you're working on, whether it be vision, text, audio, recommendation, something like that, you'll want to look into each of the PyTorch domain libraries for existing data loader or data loading functions and customizable data loading functions. So just keep that in mind. We've seen some of them. So if we go torch vision, which is what we're going to be looking at, torch vision, we've got data sets,
19:57:07
and we've got documentation, we've got data sets for each of the other domain libraries here as well. So if you're working on a text problem, it's going to be a similar set of steps to what we're going to do with our vision problem when we build food vision mini. What we have is a data set that exists somewhere. And what we want to do is bring that into PyTorch so we can build a model with it. So let's import the libraries that we need. So we're going to import torch and
19:57:35
we'll probably import an N. So we'll import that from PyTorch. And I'm just going to check the torch version here. So note, we need PyTorch 1.10.0 plus is required for this course. So if you're using Google Colab at a later date, you may have a later version of PyTorch. I'm just going to show you what version I'm using. Just going to let this load. We're going to get this ready. We're going to also set up device agnostic code right from the start this time because this is
19:58:13
best practice with PyTorch. So this way, if we have a CUDA device available, our model is going to use that CUDA device. And our data is going to be on that CUDA device. So there we go. Wonderful. We've got PyTorch 1.10.0 plus CUDA. 111. Maybe that's 11.1. So let's check if CUDA.is available. Now, I'm using Google Colab. We haven't set up a GPU yet. So it probably won't be available yet. Let's have a look. Wonderful. So because we've started a new Colab instance, it's going to use
19:58:49
the CPU by default. So how do we change that? We come up to runtime, change runtime type. I'm going to go hard there accelerator GPU. We've done this a few times now. I am paying for Google Colab Pro. So one of the benefits of that is that it our Google Colab reserves faster GPUs for you. You do don't need Google Colab Pro. As I've said to complete this course, you can use the free version, but just recall Google Colab Pro tends to give you a better GPU just because GPUs aren't free.
19:59:23
Wonderful. So now we've got access to a GPU CUDA. What GPU do I have?
19:59:30
Nvidia SMI. I have a Tesla P100 with 16 gigabytes of memory, which will be more than enough for the problem that we're going to work on in this video. So I believe that's enough to cover for the first coding video. Let's in the next section, we are working with custom datasets after all. Let's in the next video. Let's get some data, hey.
19:59:55
Now, as I said in the last video, we can't cover custom datasets without some data. So let's get some data and just remind ourselves what we're going to build. And that is food vision mini. So we need a way of getting some food images. And if we go back to Google Chrome, torch vision datasets has plenty of built-in datasets. And one of them is the food 101 dataset. Food 101. So if we go in here, this is going to take us to the original food 101 website.
20:00:30
So food 101 is 101 different classes of food. It has a challenging dataset of 101 different food categories with 101,000 images. So that's a quite a beefy dataset. And so for each class, 250 manually reviewed test images are provided. So we have per class, 101 classes, 250 testing images, and we have 750 training images. Now, we could start working on this entire dataset straight from the get go. But to practice, I've created a smaller subset of this dataset,
20:01:06
and I'd encourage you to do the same with your own problems. Start small and upgrade when necessary. So I've reduced the number of categories to three and the number of images to 10%. Now, you could reduce this to an arbitrary amount, but I've just decided three is enough to begin with and 10% of the data. And then if it works, hey, you could upscale that on your own accord. And so I just want to show you the notebook that I use to create this dataset and as extra curriculum,
20:01:38
you could go through this notebook. So if we go into extras, 04 custom data creation, this is just how I created the subset of data. So making a dataset to use with notebook number four, I created it in custom image data set or image classification style. So we have a top level folder of pizza, steak, and sushi. We have a training directory with pizza, steak, and sushi images. And we have a test directory with pizza, steak, and sushi images as well. So you can go
20:02:09
through that to check it out how it was made. But now, oh, and also, if you go to loan pytorch.io section four, there's more information here about what food 101 is. So get data. Here we go. There's all the information about food 101. There's some resources, the original food 101 data set, torch vision data sets, food 101, how I created this data set, and actually downloading the data. But now we're going to write some code, because this data set, the smaller version that I've created
20:02:40
is on the pytorch deep learning repo, under data. And then we have pizza, steak, sushi.zip. Oh, this one is a little spoiler for one of the exercises for this section. But you'll see that later. Let's go in here. Let's now write some code to get this data set from GitHub, pizza, steak, sushi.zip. And then we'll explore it, we'll become one with the data. So I just want to write down here, our data set is a subset of the food 101 data set.
20:03:14
Food 101 starts with 101 different classes of food. So we could definitely build computer vision models for 101 classes, but we're going to start smaller. Our data set starts with three classes of food, and only 10% of the images. So what's right here? And 1000 images per class, which is 750 training, 250 testing. And we have about 75 training images per class, and about 25 testing images per class. So why do this? When starting out ML projects,
20:04:05
it's important to try things on a small scale and then increase the scale when necessary.
20:04:15
The whole point is to speed up how fast you can experiment. Because there's no point trying to experiment on things that if we try to train on 100,000 images to begin with, our models might train take half an hour to train at a time. So at the beginning, we want to increase the rate that we experiment at. And so let's get some data. We're going to import requests so that we can request something from GitHub to download this
20:04:45
URL here. Then we're also going to import zip file from Python, because our data is in the form of a zip file right now. Then we're going to get path lib, because I like to use paths whenever I'm dealing with file paths or directory paths. So now let's set up a path to a data folder. And this, of course, will depend on where your data set lives, what you'd like to do. But I typically like to create a folder over here called data. And that's just going to store all of my
20:05:15
data for whatever project I'm working on. So data path equals path data. And then we're going to go image path equals data path slash pizza steak sushi. That's how we're going to have images from those three classes. Pizza steak and sushi are three of the classes out of the 101 in food 101. So if the image folder doesn't exist, so if our data folder already exists, we don't want to redownload it. But if it doesn't exist, we want to download it and unzip it. So if image path
20:05:55
is der, so we want to print out the image path directory already exists skipping download.
20:06:09
And then if it doesn't exist, we want to print image path does not exist, creating one. Beautiful. And so we're going to go image path dot mk der to make a directory. We want to make its parents if we need to. So the parent directories and we want to pass exist, okay, equals true. So we don't get any errors if it already exists. And so then we can write some code. I just want to show you what this does if we run it. So our target directory data slash pizza steak sushi does not exist.
20:06:44
It's creating one. So then we have now data and inside pizza steak sushi. Wonderful. But we're going to fill this up with some images so that we have some data to work with. And then the whole premise of this entire section will be loading this data of just images into PyTorch so that we can build a computer vision model on it. But I just want to stress that this step will be very similar no matter what data you're working with. You'll have some folder over here or maybe it'll
20:07:11
live on the cloud somewhere. Who knows wherever your data is, but you'll want to write code to load it from here into PyTorch. So let's download pizza steak and sushi data. So I'm going to use width. I'll just X over here. So we have more screen space with open. I'm going to open the data path slash the file name that I'm trying to open, which will be pizza steak sushi dot zip. And I'm going to write binary as F. So this is essentially saying I'm doing this in advance because I know
20:07:47
I'm going to download this folder here. So I know the the file name of it, pizza steak sushi dot zip. I'm going to download that into Google collab and I want to open it up. So request equals request dot get. And so when I want to get this file, I can click here. And then if I click download, it's going to what do you think it's going to do? Well, let's see. If I wanted to download it locally, I could do that. And then I could come over here. And then I could click upload if I
20:08:25
wanted to. So upload the session storage. I could upload it from that. But I prefer to write code so that I could just run this cell over again and have the file instead of being download to my local computer. It just goes straight into Google collab. So to do that, we need the URL from here. And I'm just going to put that in there. It needs to be as a string.
20:08:49
Excuse me. I'm getting trigger happy on the shift and enter. Wonderful. So now I've got a request to get the content that's in here. And GitHub can't really show this because this is a zip file of images, spoiler alert. Now let's keep going. We're going to print out that we're downloading pizza, stake and sushi data dot dot dot. And then I'm going to write to file the request dot content. So the content of the request that I just made to GitHub. So that's request is here.
20:09:26
Using the Python request library to get the information here from GitHub. This URL could be wherever your file has been stored. And then I'm going to write the content of that request to my target file, which is this. This here. So if I just copy this, I'm going to write the data to here data path slash pizza, stake sushi zip. And then because it's a zip file, I want to unzip it. So unzip pizza, stake sushi data. Let's go with zip file. So we imported zip file up there,
20:10:03
which is a Python library to help us deal with zip files. We're going to use zip file dot zip file. We're going to pass it in the data path. So just the path that we did below, data path slash pizza, stake sushi dot zip. And this time, instead of giving it right permissions, so that's what wb stands for, stands for right binary. I'm going to give it read permissions. So I want to read this target file instead of writing it. And I'm going to go as zip ref.
20:10:36
We can call this anything really, but zip ref is kind of, you'll see this a lot in different Python examples. So we're going to print out again. So unzipping pizza, stake, and sushi data. Then we're going to go zip underscore ref dot extract all. And we're going to go image path. So what this means is it's taking the zip ref here. And it's extracting all of the information that's within that zip ref. So within this zip file, to the image path,
20:11:14
which is what we created up here. So if we have a look at image path, let's see that.
20:11:22
Image path. Wonderful. So that's where all of the contents of that zip file are going to go into this file. So let's see it in action. You're ready. Hopefully it works. Three, two, one, run. File is not a zip file. Oh, no, what do we get wrong? So did I type this wrong?
20:11:47
Got zip data path. Oh, we got the zip file here. Pizza, stake, sushi, zip, read data path.
20:11:59
Okay, I found the error. So this is another thing that you'll have to keep in mind. And I believe we've covered this before, but I like to keep the errors in these videos so that you can see where I get things wrong, because you never write code right the first time. So we have this link in GitHub. We have to make sure that we have the raw link address. So if I come down to here and copy the link address from the download button, you'll notice a slight
20:12:24
difference if we come back into here. So I'm just going to copy that there. So if we step through this GitHub, Mr. D Burke pytorch deep learning, we have raw instead of blob. So that is why we've had an error is that our code is correct. It's just downloading the wrong data. So let's change this to the raw. So just keep that in mind, you must have raw here. And so let's see if this works.
20:12:52
Do we have the correct data? Oh, we might have to delete this. Oh, there we go. Test. Beautiful. Train. Pizza steak sushi. Wonderful. So it looks like we've got some data. And if we open this up, what do we have? We have various JPEGs. Okay. So this is our testing data. And if we click on there, we've got an image of pizza. Beautiful. So we're going to explore this a little bit more in the next video. But that is some code that we've written to download data sets
20:13:21
or download our own custom data set. Now, just recall that we are working specifically on a pizza steak and sushi problem for computer vision. However, our whole premise is that we have some custom data. And we want to convert these. How do we get these into tenses? That's what we want to do. And so the same process will be for your own problems. We'll be loading a target data set and then writing code to convert whatever the format the data set is in into tenses for PyTorch.
20:13:52
So I'll see you in the next video. Let's explore the data we've downloaded.
20:14:00
Welcome back. In the last video, we wrote some code to download a target data set, our own custom data set from the PyTorch deep learning data directory. And if you'd like to see how that data set was made, you can go to PyTorch deep learning slash extras. It's going to be in the custom data creation notebook here for 04. So I've got all the code there. All we've done is take data from the food 101 data set, which you can download from this website here, or from torch
20:14:31
vision. So if we go to torch vision, food 101. We've got the data set built into PyTorch there. So I've used that data set from PyTorch and broken it down from 101 classes to three classes so that we can start with a small experiment. So there we go. Get the training data, data sets food 101, and then I've customized it to be my own style. So if we go back to CoLab, we've now got pizza steak sushi, a test folder, which will be our testing images, and a train folder,
20:15:04
which will be our training images. This data is in standard image classification format. But we'll cover that in a second. All we're going to do in this video is kick off section number two, which is becoming one with the data, which is one of my favorite ways to refer to data preparation and data exploration. So we're coming one with the data. And I'd just like to show you one of my favorite quotes from Abraham loss function. So if I had eight hours to build a machine learning model,
20:15:41
I'd spend the first six hours preparing my data set. And that's what we're going to do. Abraham loss function sounds like he knows what is going on. But since we've just downloaded some data, let's explore it. Hey, and we'll write some code now to walk through each of the directories. How you explore your data will depend on what data you've got. So we've got a fair few different directories here with a fair few different folders within them. So how about we walk through each
20:16:12
of these directories and see what's going on. If you have visual data, you probably want to visualize an image. So we're going to do that in the second two, write a little doc string for this helper function. So walks through the path, returning its contents. Now, just in case you didn't know Abraham loss function does not exist as far as I know. But I did make up that quote. So we're going to use the OS dot walk function, OS dot walk. And we're going to pass it in a dirt path. And
20:16:46
what does walk do? We can get the doc string here. Directory tree generator. For each directory in the directory tree rooted at the top, including top itself, but in excluding dot and dot dot, yields a three tuple, derpath, der names, and file names. You can step through this in the Python documentation, if you'd like. But essentially, it's just going to go through our target directory, which in this case will be this one here. And walk through each of these directories printing out
20:17:17
some information about each one. So let's see that in action. This is one of my favorite things to do if we're working with standard image classification format data. So there are lane, length, der names, directories. And let's go land, land, file names. We say at length, like I've got the G on the end, but it's just land images in, let's put in here, derpath. So a little bit confusing if you've never used walk before, but it's so exciting to see all of the information in all
20:17:54
of your directories. Oh, we didn't read and run it. Let's check out function now walk through der. And we're going to pass it in the image path, which is what? Well, it's going to show us. How beautiful. So let's compare what we've got in our printout here. There are two directories and zero images in data, pizza, steak sushi. So this one here, there's zero images, but there's two directories test and train wonderful. And there are three directories in data, pizza, steak, sushi,
20:18:24
test. Yes, that looks correct. Three directories, pizza, steak, sushi. And then we have zero directories and 19 images in pizza, steak, sushi, slash test, steak. We have a look at this. So that means there's 19 testing images for steak. Let's have a look at one of them. There we go. Now, again, these are from the food 101 data set, the original food 101 data set, which is just a whole bunch of images of food, 100,000 of them. There's some steak there. Wonderful. And we're trying to
20:18:55
build a food vision model to recognize what is in each image. Then if we jump down to here, we have three directories in the training directory. So we have pizza, steak, sushi. And then we have 75 steak images, 72 sushi images and 78 pizza. So slightly different, but very much the same numbers. They're not too far off each other. So we've got about 75 or so training images, and we've got about 25 or so testing images per class. Now these were just randomly selected
20:19:26
from the food 101 data set 10% of three different classes. So let's keep pushing forward. And we're going to set up our training and test parts. So I just want to show you, we'll just set up this, and then I'll just show you the standard image classification setup, image path.train. And we're going to go tester. So if you're working on image classification problem, we want to set this up as test. And then if we print out the trainer and the tester, this is what we're going to be
20:20:04
trying to do. We're going to write some code to go, Hey, look at this path for our training images. And look at this path for our testing images. And so this is the standard image classification data format is that you have your overall data set folder. And then you have a training folder dedicated to all of the training images that you might have. And then you have a testing folder dedicated to all of the testing images that you might have. And you could have a validation
20:20:31
data set here as well if you wanted to. But to label each one of these images, the class name is the folder name. So all of the pizza images live in the pizza directory, the same for steak, and the same for sushi. So depending on your problem, your own data format will depend on whatever you're working on, you might have folders of different text files or folders of different audio files. But the premise remains, we're going to be writing code to get our data here
20:21:04
into tenses for use with PyTorch. And so where does this come from? This image data classification format. Well, if we go to the torch vision dot data sets documentation, as you start to work with more data sets, you'll start to realize that there are standardized ways of storing specific types of data. So if we come down to here, base classes for custom data sets, we'll be working towards using this image folder data set. But this is a generic data
20:21:33
loader where the images are arranged in this way by default. So I've specifically formatted our data to mimic the style that this pre built data loading function is for. So we've got a root directory here in case of we were classifying dog and cat images, we have root, then we have a dog folder, then we have various images. And the same thing for cat, this would be dog versus cat. But the only difference for us is that we have food images, and we have pizza steak sushi. If we wanted to use the
20:22:06
entire food 101 data set, we would have 101 different folders of images here, which is totally possible. But to begin with, we're keeping things small. So let's keep pushing forward. As I said, we're dealing with a computer vision problem. So what's another way to explore our data, other than just walking through the directories themselves. Let's visualize an image, hey? But we've done that before with just clicking on the file. How about we write some code to do so.
20:22:35
We'll replicate this but with code. I'll see you in the next video.
20:22:42
Welcome back. In the last video, we started to become one with the data. And we learned that we have about 75 images per training class and about 25 images per testing class. And we also learned that the standard image classification data structure is to have the steak images within the steak folder of the training data set and the same for test, and the pizza images within the pizza folder, and so on for each different image classification class that we might have.
20:23:14
So if you want to create your own data set, you might format it in such a way that your training images are living in a directory with their classification name. So if you wanted to classify photos of dogs and cats, you might create a training folder of train slash dog train slash cat, put images of dogs in the dog folder, images of cats in the cat folder, and then the same for the testing data set. But the premise remains, I'm going to sound like a broken record here.
20:23:42
We want to get our data from these files, whatever files they may be in, whatever data structure they might be in, into tenses. But before we do that, let's keep becoming one with the data. And we're going to visualize an image. So visualizing an image, and you know how much I love randomness. So let's select a random image from all of the files that we have in here. And let's plot it, hey, because we could just click through them and visualize them. But I like to do things with
20:24:11
code. So specifically, let's let's plan this out. Let's write some code to number one is get all of the image paths. We'll see how we can do that with the path path lib library. We then want to pick a random image path using we can use Python's random for that. Python's random dot choice will pick a single image random dot choice. Then we want to get the image class name. And this is where part lib comes in handy. Class name, recall that whichever target image we pick, the class name will
20:24:51
be whichever directory that it's in. So in the case of if we picked a random image from this directory, the class name would be pizza. So we can do that using, I think it's going to be path lib dot path. And then we'll get the parent folder, wherever that image lives. So the parent image parent folder that parent directory of our target random image. And we're going to get the stem of that. So we have stem, stem is the last little bit here. Number four, what should we do? Well,
20:25:23
we want to open the image. So since we're working with images, let's open the image with Python's pill, which is Python image library, but we'll actually be pillow. So if we go Python pillow, a little bit confusing when I started to learn about Python image manipulation. So pillow is a friendly pill for, but it's still called pill. So just think of pillow as a way to process images with Python. So pill is the Python imaging library by Frederick Lund. And so Alex Clark and
20:26:01
contributors have created pillow. So thank you, everyone. And let's go to number five. What do we want to do as well? We want to, yeah, let's get some metadata about the image. We'll then show the image and print metadata. Wonderful. So let's import random, because machine learning is all about harnessing the power of randomness. And I like to use randomness to explore data as well as model it. So let's set the seed. So we get the same image on both of our ends. So random dot seed.
20:26:38
I'm going to use 42. You can use whatever you'd like. But if you'd like to get the same image as me, I'd suggest using 42 as well. Now let's get all the image paths. So we can do this because our image path list, we want to get our image path. So recall that our image path
20:27:02
is this. So this folder here, I'm just going to close all this. So this is our image path, this folder here, you can also go copy path if you wanted to, we're just going to get something very similar there. That's going to error out. So I'll just comment that. So it doesn't error. That's our path. But we're going to keep it in the POSIX path format. And we can go list. Let's create a list of image path dot glob, which stands for grab. I don't actually know what glob stands
20:27:34
for. But to me, it's like glob together. All of the images that are all of the files that suit a certain pattern. So glob together for me means stick them all together. And you might be able to correct me if I've got the wrong meaning there. I'd appreciate that. And so we're going to pass in a certain combination. So we want star slash star. And then we want star dot jpg. Now why are we doing this? Well, because we want every image path. So star is going to be this first
20:28:10
directory here. So any combination, it can be train or test. And then this star means anything for what's inside tests. And let's say this first star is equal to test. This second star is equal to anything here. So it could be any of pizza, steak or sushi. And then finally, this star, let's say it was test pizza. This star is anything in here. And that is before dot jpg. So it could be any one of these files here. Now this will make more sense once we print it out.
20:28:42
So image path list, let's have a look. There we go. So now we've got a list of every single image that's within pizza steak sushi. And this is just another way that I like to visualize data is to just get all of the paths and then randomly visualize it, whether it be an image or text or audio, you might want to randomly listen to it. Recall that each each of the domain libraries have different input and output methods for different data sets. So if we come to torch vision, we have
20:29:13
utils. So we have different ways to draw on images, reading and writing images and videos. So we could load an image via read image, we could decode it, we could do a whole bunch of things. I'll let you explore that as extra curriculum. But now let's select a random image from here and plot it. So we'll go number two, which was our step up here, pick a random image. So pick a random image path. Let's get rid of this. And so we can go random image path equals random
20:29:51
dot choice, harness the power of randomness to explore our data. Let's get a random image from image path list, and then we'll print out random image path, which one was our lucky image that we selected. Beautiful. So we have a test pizza image is our lucky random image. And
20:30:13
because we've got a random seed, it's going to be the same one each time. Yes, it is. And if we comment out the random seed, we'll get a different one each time. We've got a stake image. We've got another stake image. Another stake image. Oh, three in a row, four in a row. Oh, pizza. Okay, let's keep going. So we'll get the image class
20:30:36
from the path name. So the image class is the name of the directory, because our image data is in standard image classification format, where the image is stored. So let's do that image class equals random image path dot parent dot stem. And then we're going to print image class. What do we get? So we've got pizza. Wonderful. So the parent is this folder here. And then the stem is the end of that folder, which is pizza. Beautiful. Well, now what are we up to now? We're working with
20:31:17
images. Let's open up the image so we can open up the image using pill. We could also open up the image with pytorch here. So with read image, but we're going to use pill to keep things a little bit generic for now. So open image, image equals image. So from pill import image, and the image class has an open function. And we're just going to pass it in here, the random image path. Note if this is corrupt, if your images corrupt, this may error. So then you could potentially use this
20:31:48
to clean up your data set. I've imported a lot of images with image dot open of our target data set here. I don't believe any of them are corrupt. But if they are, please let me know. And we'll find out later on when our model tries to train on it. So let's print some metadata. So when we open our image, we get some information from it. So let's go our random image path is what? Random image path. We're already printing this out, but we'll do it again anyway. And then we're going to go the image
20:32:22
class is equal to what will be the image class. Wonderful. And then we can print out, we can get some metadata about our images. So the image height is going to be IMG dot height. We get that metadata from using the pill library. And then we're going to print out image width. And we'll get IMG dot width. And then we'll print the image itself. Wonderful. And we can get rid of this, and we can get rid of this. Let's now have a look at some random images from our data set.
20:32:59
Lovely. We've got an image of pizza there. Now I will warn you that the downsides of working with food data is it does make you a little bit hungry. So there we've got some sushi. And then we've got some more sushi. Some steak. And we have a steak, we go one more for good luck. And we finish off with some sushi. Oh, that could be a little bit confusing to me. I thought that might be steak to begin with. And this is the scene. Now we'll do one more. Why it's important to sort of visualize
20:33:31
your images randomly, because you never know what you're going to come across. And this way, once we visualize enough images, you could do this a hundred more times. You could do this 20 more times until you feel comfortable to go, Hey, I feel like I know enough about the data now. Let's see how well our model goes on this sort of data. So I'll finish off on this steak image. And now I'll set your little challenge before the next video is to visualize an image like we've
20:33:56
done here. But this time do it with matplotlib. So try to visualize an image with matplotlib. That's your little challenge before the next video. So give that a go. We want to do a random image as well. So quite a similar set up to this. But instead of printing out things like this, we want to visualize it using matplotlib. So try that out and we'll do it together in the next video.
20:34:24
Oh, we are well on the way to creating our own PyTorch custom data set. We've started to become one with the data. But now let's continue to visualize another image. I set you the challenge in the last video to try and replicate what we've done here with the pill library with matplotlib. So now let's give it a go. Hey, and why use matplotlib? Well, because matplotlib and I'm going to import numpy as well, because we're going to have to convert this image into an array. That was a
20:34:53
little trick that I didn't quite elaborate on. But I hope you tried to decode it out and figure it out from the errors you received. But matplotlib is one of the most fundamental data science libraries. So you're going to see it everywhere. So it's just important to be aware of how to plot images and data with matplotlib. So turn the image into an array. So we can go image as array. And I'm going to use the numpy method NP as array. We're going to pass it in the image, recall that
20:35:29
the image is the same image that we've just set up here. And we've already opened it with pill.
20:35:36
And then I'm going to plot the image. So plot the image with matplotlib. plt.figure. And then we can go fig size equals 10, seven. And then we're going to go plt.im show image as array, pass it in the array of numbers. I'm going to set the title here as an f string. And then I'm going to pass in image class, equals image class. Then I'm going to pass in image shape. So we can get the shape here. Now this is another important thing to be aware of of your different
20:36:15
datasets when you're exploring them is what is the shape of your data? Because what's one of the main errors in machine learning and deep learning? It's shape mismatch issues. So if we know the shape of our data where we can start to go, okay, I kind of understand what shape I need my model layers to be in what what shape I need my other data to be in. And I'm going to turn the axes off here. Beautiful. So look at what we've got. Now I've just thrown this in here without really
20:36:44
explaining it. But we've seen this before in the computer vision section. As our image shape is 512 3063. Now the dimensions here are height is 512 pixels. The width is 306 pixels. And it has three color channels. So what format is this? This is color channels last, which is the default for the pill library. There's also the default for map plot lib. But pytorch recall is default if we put the color channels at the start color channels first. Now there is a lot of debate as
20:37:20
I've said over which is the best order. It looks like it's leading towards going towards this. But for now pytorch defaults to color channels first. But that's okay. Because we can manipulate these dimensions to what we need for whatever code that we're writing. And the three color channels is what red, green and blue. So if you combine red, green and blue in some way, shape or form, you get the different colors here that represent our image. And so if we have a look at our image
20:37:47
as a ray. Our image is in numerical format. Wonderful. So okay. We've got one way to do this for one image. I think we start moving towards scaling this up to do it for every image in our data folder. So let's just finish off this video by visualizing one more image. What do we get? Same premise. The image is now as an array, different numerical values. We've got a delicious looking pizza here of shave 512 512 with color channels last. And we've got the same thing up here. So
20:38:28
that is one way to become one with the data is to visualize different images, especially random images. You could do the same thing visualizing different text samples that you're working with or listening to different audio samples. It depends what domain you're working in. So now in the next video, let's start working towards turning all of the images in here. Now that we visualize some of them and become one with the data, we've seen that the shapes are varying in terms of
20:38:54
height and width. But they all look like they have three color channels because we have color images. But now we want to write code to turn all of these images into pytorch tenses. So let's start moving towards that. I'll see you in the next video.
20:39:12
Hello and welcome back. In the last video, we converted an image to a NumPy array. And we saw how an image can be represented as an array. But what if we'd like to get this image from our custom data set over here, pizza steak sushi into pytorch? Well, let's cover that in this video. So I'm going to create a new heading here. And it's going to be transforming data. And so what we'd like to do here is I've been hinting at the fact the whole time is we want
20:39:45
to get our data into tensor format, because that is the data type that pytorch accepts. So let's write down here before we can use our image data with pytorch. Now this goes for images, other vision data, it goes for text, it goes to audio, basically whatever kind of data set you're working with, you need some way to turn it into tenses. So that's step number one. Turn your target data into tenses. In our case, it's going to be a numerical representation of our images.
20:40:24
And number two is turn it into a torch dot utils dot data dot data set. So recall from a previous video that we've used the data set to house all of our data in tensor format. And then subsequently, we've turned our data sets, our pytorch data sets into torch dot utils dot data dot data loader. And a data loader creates an iterable or a batched version of our data set. So for short, we're going to call these data set and data loader. Now, as I discussed previously, if we go to the pytorch
20:41:11
documentation torch vision for torch vision, this is going to be quite similar for torch audio torch text, torch rec torch data eventually when it comes out of beta, there are different ways to create such data sets. So we can go into the data sets module, and then we can find built-in data sets, and then also base classes for custom data sets. But if we go into here, image folder, there's another parameter I'd like to show you, and this is going to be universal across many of
20:41:47
your different data types is the transform parameter. Now, the transform parameter is a parameter we can use to pass in some transforms on our data. So when we load our data sets from an image folder, it performs a transform on those data samples that we've sent in here as the target data folder. Now, this is a lot more easier to understand through illustration, rather than just talking about it. So let's create a transform. And the main transform we're going to be doing is
20:42:20
transforming our data, and we're turning it into tenses. So let's see what that looks like. So we're going to just going to re import all of the main libraries that we're going to use. So from torch utils dot data, let's import data loader. And we're going to import from torch vision. I'm going to import data sets. And I'm also going to import transforms. Beautiful. And I'm going to create another little heading here, this is going to be 3.1, transforming data with torch vision dot
20:42:55
transform. So the main transform we're looking to here is turning out images from JPEGs.
20:43:04
If we go into train, and then we go into any folder, we've got JPEG images.
20:43:09
And we want to turn these into tensor representation. So there's some pizza there. We'll get out of this. Let's see what we can do. How about we create a transform here, write a transform for image. And let's start off by calling it data transform. And I'm going to show you how we can combine a few transforms together. If you want to combine transforms together, you can use transforms dot compose. You can also use
20:43:38
an n dot sequential to combine transforms. But we're going to stick with transforms dot compose for now. And it takes a list. And so let's just write out three transforms to begin with. And then we can talk about them after we do so. So we want to resize our images
20:43:59
to 6464. Now, why might we do this? Well, do you recall in the last section computer vision, we use the tiny VGG architecture. And what size were the images that the tiny VGG architecture took?
20:44:14
Well, we replicated the CNN website version or the CNN explainer website version, and they took images of size 6464. So perhaps we want to leverage that computer vision model later on. So we're going to resize our images to 6464. And then we're going to create another transform. And so this is, I just want to highlight how transforms can help you manipulate your data in a certain way. So if we wanted to flip the images, which is a form of data augmentation, in other
20:44:42
words, artificially increasing the diversity of our data set, we can flip the images randomly on the horizontal. So transforms dot random horizontal flip. And I'm going to put a probability in here of p equals 0.5. So that means 50% of the time, if an image goes through this transform pipeline, it will get flipped on the horizontal axis. As I said, this makes a lot more sense when we visualize it. So we're going to do that very shortly. And finally, we're going to turn the image into
20:45:19
a torch tensor. So we can do this with transforms dot to tensor. And now where might you find such transforms? So this transform here says to tensor, if we have a look at the doc string, we got convert a pill image, which is what we're working with right now, or a NumPy array to a tensor. This transform does not support torch script. If you'd like to find out what that is, I'd like to read the documentation for that. It's essentially turning your pytorch code into a
20:45:51
Python script. It converts a pill image or a NumPy array from height with color channels in the range 0 to 255, which is what our values are up here. They're from 0 to 255, red, green and blue, to a torch float tensor of shape color channels height width in the range 0 to 1. So it will take our tensor values here or our NumPy array values from 0 to 255 and convert them into a torch tensor in the range 0 to 1. We're going to see this later on in action. But this is our first
20:46:27
transform. So we can pass data data through that. In fact, I'd encourage you to try that out. See what happens when you pass in data transform. What happens when you pass it in our image as a ray? Image as a ray. Let's see what happens. Hey, oh, image should be pill image got class NumPy array. What if we just pass in our straight up image? So this is a pill image. There we go. Beautiful. So if we look at the shape of this, what do we get?
20:47:01
3 64 64. There's 64. And if what if we wanted to change this to 224, which is another common value for
20:47:11
computer vision models to 24 to 24. Do you see how powerful this is? This little transforms module, the torch vision library will change that back to 64 64. And then if we have a look at what D type of our transform tensor is, we get torch float 32. Beautiful. So now we've got a way to transform our images into tensors. And so, but we're still only doing this with one image. How about we progress towards doing it for every image in our data folder here?
20:47:44
But before we do that, I'd like to visualize what this looks like. So in the next video, let's write some code to visualize what it looks like to transform multiple images at a time. And I think it'd be a good idea to compare the transform that we're doing to the original image. So I'll see you in the next video. Let's write some visualization code.
20:48:06
Let's now follow our data explorer's motto of visualizing our transformed images. So we saw what it looks like to pass one image through a data transform. And if we wanted to find more documentation on torch vision transforms, where could we go? There is a lot of these. So transforming and augmenting images, this is actually going to be your extra curriculum for this video. So transforms are common image transformations available in the transforms module. They can be chained together
20:48:36
using compose, which is what we've already done. Beautiful. And so if you'd like to go through all of these, there's a whole bunch of different transforms that you can do, including some data augmentation transforms. And then if you'd like to see them visually, I'd encourage you to check out illustration of transforms. But let's write some code to explore our own transform visually first. So I'll leave this as a link. So I'm going up here, right here, transforms
20:49:06
help you get your images ready to be used with a model slash perform data augmentation. Wonderful. So we've got a way to turn images into tenses. That's what we want for our model. We want our images as pytorch tenses. The same goes for any other data type that you're working with. But now I'd just like to visualize what it looks like if we plot a number of transformed images. So we're going to make a function here that takes in some image paths, a transform,
20:49:41
a number of images to transform at a time and a random seed here, because we're going to harness the power of randomness. And sometimes we want to set the seed. Sometimes we don't. So we have an image path list that we've created before, which is just all of the image paths that we have of our data set. So data, pizza, steak sushi. Now how about we select some random image paths and then take the image from that path, run it through our data transform, and then compare the
20:50:15
original image of what it looks like and the transformed image and what that looks like.
20:50:22
Let's give it a try, hey? So I'm going to write a doc string of what this does, and then selects random images from a path of images and loads slash transforms them, then plots the original verse, the transformed version. So that's quite a long doc string, but that'll be enough. We can put in some stuff for the image paths, transforms, and seed. We'll just code this out. Let's go random seed, we'll create the seed. Maybe we do it if seed, random seed.
20:51:00
Let's put that, and we'll set seed to equal none by default. That way we can, we'll see if this works, hey, if in doubt, coded out random image paths, and then we're going to go random sample from the image paths and the number of sample that we're going to do. So random sample is going to, this will be a list on which part in here that this is a list. So we're going to randomly sample k, which is going to be n. So three images from our image path list. And then we're going to go for
20:51:34
image path, we're going to loop through the randomly sampled image parts. You know how much I love harnessing the power of randomness for visualization. So for image path in random image paths, let's open up that image using pill image dot open image path as f. And then we're going to create a figure and an axes. And we're going to create a subplot with my plot lib. So subplots. And we want it to create one row. So it goes n rows and calls. One row and n calls equals two. And then
20:52:13
on the first or the zeroth axis, we're going to plot the original image. So in show, we're just going to pass it straight in f. And then if we want to go x zero, we're going to set the title. So set title, we're going to set it to be the original. So we'll create this as an f string, original, and then new line will create a size variable. And this is going to be f dot size. So we're just getting the size attribute from our file. So we'll keep going, and we'll turn off the axes here.
20:52:48
So axis, and we're going to set that to false. Now let's transform on the first axes plot. We're going to transform and plot target image. This is so that our images are going to be side by side, the original and the transformed version. So there's one thing that we're going to have to do. I'll just, I'll code it out in a wrong way first. I think that'll be a good way to illustrate what's going on. f. So I'm just going to put a note here. Note, we will need to change shape for
20:53:23
matplotlib, because we're going to come back here. Because what does this do? What have we noticed that our transform does? If we check the shape here, oh, excuse me, it converts our image to color channels first. Whereas matplotlib prefers color channels last. So just keep that in mind for when we're going forward. This code, I'm writing it, it will error on purpose. So transformed image. And then we're going to go axe one as well. We're going to set the title,
20:53:58
which is going to be transformed. And then we'll create a new line and we'll say size is going to be
20:54:07
transformed image dot shape. Or probably a bit of, yeah, we could probably go shape here. And then finally, we're going to go axe one, we're going to turn the axis, we're going to set that to false. You can also set it to off. So you could write false, or you could write off, you might see that different versions of that somewhere. And I'm going to write a super title here, which we'll see what this looks like class is going to be image path. So we're getting the target image path. And we're
20:54:41
just going to get the attribute or the parent attribute, and then the stem attribute from that, just like we did before, to get the class name. And then I'm going to set this to a larger font size, so that we make some nice looking plots, right? If we're going to visualize our data, we might as well make our plots visually appealing. So let's plot some transformed data or transformed images. So image paths, we're going to set this to image part list, which is just the variable we
20:55:08
have down below, which is the part list, a list containing all of our image paths. Our transform, we're going to set our transform to be equal to our data transform. So this just means that if we pass the transform in, our image is going to go through that transform, and then go through all of these is going to be resized, it's going to be randomly horizontally flipped, and it's going to be converted to a tensor. And then so we're going to set that data transfer there or data transform,
20:55:37
sorry, and is going to be three. So we plot three images, and we'll set the seed to 42 to begin with. Let's see if this works. Oh, what did we get wrong? We have invalid shape. As I said, I love seeing this error, because we have seen this error many times, and we know what to do with it. We know that we have to rearrange the shapes of our data in some way, shape or form. Wow, I said shape a lot there. That's all right. Let's go here, permute. This is what we have to do. We have to permute,
20:56:07
we have to swap the order of the axes. So right now, our color channels is first. So we have to bring this color channel axis or dimension to the end. So we need to shuffle these across. So 64 into here, 64 into here, and three on the end. We need to, in other words, turn it from color channels first to color channels last. So we can do that by permuting it to have the first axis come now in the zero dimension spot. And then number two was going to be in the first
20:56:35
dimension spot. And then number zero was going to be at the back end. So this is essentially going from C H W, and we're just changing the order to be H W C. So the exact same data is going to be within that tensor. We're just changing the order of the dimensions. Let's see if this works.
20:57:00
Look at that. Oh, I love seeing some manipulated data. We have a class of pizza and the original image is there, and it's 512 by 512. But then we've resized it using our transform. Notice that it's a lot more pixelated now, but that makes sense because it's only 64 64 pixels. Now, why might we do such a thing? Well, one, if is this image still look like that? Well, to me, it still does. But the most important thing will be does it look like that to our model? Does it still look
20:57:29
like the original to our model? Now 64 by 64, there is less information encoded in this image. So our model will be able to compute faster on images of this size. However, we may lose some performance because not as much information is encoded as the original image. Again, the size of an image is something that you can control. You can set it to be a hyper parameter. You can tune the size to see if it improves your model. But I've just decided to go 60 64 64 3 in line
20:58:01
with the CNN explainer website. So a little hint, we're going to be re replicating this model that we've done before. Now you notice that our images are now the same size 64 64 3 as what the CNN explainer model uses. So that's where I've got that from. But again, you could change this to size to whatever you want. And we see, oh, we've got a stake image here. And you notice that our image has been flipped on the horizontal. So the horizontal access, our image has just been flipped
20:58:30
same with this one here. So this is the power of torch transforms. Now there are a lot more transforms, as I said, you can go through them here to have a look at what's going on. Illustrations of transforms is a great place. So there's resize, there's center crop, you can crop your images, you can crop five different locations, you can do grayscale, you can change the color, a whole bunch of different things. I'd encourage you to check this out. That's your extra curriculum
20:59:02
for this video. But now that we've visualized a transform, this is what I hinted at before that we're going to use this transform for when we load all of our images in, using into a torch data set. So I just wanted to make sure that they had been visualized first. We're going to use our data transform in the next video when we load all of our data using a torch vision dot data sets helper function. So let's give that a go. I'll see you in the next video.
20:59:38
Have a look at that beautiful plot. We've got some original images and some transformed images. And the beautiful thing about our transformed images is that they're in tensor format, which is what we need for our model. That's what we've been slowly working towards. We've got a data set. And now we've got a way to turn it into tensors ready for a model. So let's just visualize what another, I'll turn the seed off here so we can look at some more random
21:00:03
images. There we go. Okay, so we've got stake pixelated because we're downsizing 64, 64, 3. Same thing for this one. And it's been flipped on the horizontal. And then same thing for this pizza image and we'll do one more to finish off. Wonderful. So that is the premise of transforms turning our images into tensors and also manipulating those images if we want to. So let's get rid of this. I'm going to make another heading. We're up to section or part four now.
21:00:38
And this is going to be option one. So loading image data using image folder. And now I'm going to turn that into markdown. And so let's go torch vision data sets. So recall how each one of the torch vision domain libraries has its own data sets module that has built in functions for helping you load data. In this case, we have an image folder. And there's a few others here if you'd like to look into those. But an image folder, this class is going to help us load in data that
21:01:16
is in this format, the generic image classification format. So this is a prebuilt data sets function. Just like there's prebuilt data sets, we can use prebuilt data set functions. Now option two later on, this is a spoiler, is we're going to create our own custom version of a data set loader. But we'll see that in a later video. So let's see how we can use image folder to load all of our custom data, our custom images into tensors. So this is where the transform is going to come in
21:01:48
helpful. So let's write here, we can load image classification data using, let's write this, let's write the full path name, torch vision dot data sets dot image folder. Put that in there,
21:02:07
beautiful. And so let's just start it out, use image folder to create data sets. Now in a previous video, I hinted at the fact that we can pass a transform to our image folder class. That's going to be right here. So let's see what that looks like in practice. So from torch vision, I'm going to import data sets, because that's where the image folder module lives. And then we can go train data equals data sets dot image folder. And we're going to pass in the root, which is our train
21:02:43
der, because we're going to do it for the training directory first. And then we're going to pass in a transform, which is going to be equal to our data transform. And then we're going to pass in a target transform, but we're going to leave this as none, which is the default, I believe, we go up to here. Yeah, target transform is optional. So what this means is this is going to be a transform for the data. And this is going to be a transform for the label slash target.
21:03:17
PyTorch likes to use target, I like to use label, but that's okay. So this means that we don't need a target transform, because our labels are going to be inferred by the target directory where the images live. So our pizza images are in this directory, and they're going to have pizza as the label, because our data set is in standard image classification format. Now, if your data set wasn't in a standard image classification format, you might use a different data loader here. A lot of them
21:03:47
will have a transform for the data. So this transform is going to run our images, whatever images are loaded from these folders, through this transform that we've created here, it's going to resize them, randomly flip them on the horizontal, and then turn them into tenses, which is exactly how we want them for our PyTorch models. And if we wanted to transform the labels in some way, shape or form, we could pass in a target transform here. But in our case, we don't need to transform the labels.
21:04:18
So let's now do the same thing for the test data. And so that's why I wanted to visualize our transforms in the previous videos, because otherwise we're just passing them in as a transform. So really, what's going to happen behind the scenes is all of our images are going to go through these steps. And so that's what they're going to look like when we turn them into a data set. So let's create the test data here or the test data set. The transform, we're going to
21:04:45
transform the test data set in the same way we've transformed our training data set. And we're just going to leave that like that. So let's now print out what our data sets look like, train data, and test data. Beautiful. So we have a data set, a torch data set, which is an image folder. And we have number of data points. This is going to be for the training data set. We have 225. So that means about 75 images per class. And we have the root location,
21:05:17
which is the folder we've loaded them in from, which is our training directory. We've set these two up before, trained and tester. And then we have a transform here, which is a standard transform, a resize, followed by random horizontal flip, followed by two tensor. Then we've got basically the same output here for our test directory, except we have less samples there. So let's get a few little attributes from the image folder. This is one of the benefits of using a pytorch prebuilt
21:05:49
data loader, is that or data set loader is that it comes with a fair few attributes. So we could go to the documentation, find this out from in here, inherits from data set folder, keep digging into there, or we could just come straight into Google collab. Let's go get class names as a list. Can we go train data dot and then press tab? Beautiful. So we've got a fair few things here that are attributes. Let's have a look at classes. This is going to give us a list of the class names,
21:06:20
class names. This is very helpful later on. So we've got pizza steak sushi. We're trying to do everything with code here. So if we have this attribute of train data dot classes, we can use this list later on for when we plot images straight from our data set, or make predictions on them and we want to label them. You can also get class names as a dictionary, map to their integer index, that is, so we can go train data dot and press tab. We've got class
21:06:54
to ID X. Let's see what this looks like. Class decked. Wonderful. So then we've got our string class names mapped to their integer. So we've got pizza is zero, steak is one, sushi is two. Now, this is where the target transform would come into play. If you wanted to transform those
21:07:16
these labels here in some way, shape or form, you could pass a transform into here. And then if we keep going, let's check the lengths of what's going on. Check the lengths of our data set. So we've seen this before, but this is going to just give us how many samples that we have length, train data, length, test data, beautiful. And then of course, if you'd like to explore more attributes, you can go train data dot, and then we've got a few other things,
21:07:44
functions, images, loader, samples, targets. If you wanted to just see the images, you can go dot samples. If you wanted to see just the labels, you can go dot targets. This is going to be all of our labels. Look at that. And I believe they're going to be an order. So we're going to have zero, zero, zero, one, one, one, two, two, and then if we wanted to have a look, let's say we have a look at the first sample, hey, we have data, pizza, steak sushi, train, pizza. There's the image path,
21:08:15
and it's a label zero for pizza. Wonderful. So now we've done that. How about we, we've been visualizing this whole time. So let's keep up that trend. And let's visualize a sample and a label from the train data data set. So in this video, we've used image folder to load our images into tenses. And because our data is already in standard image classification format, we can use one of torch vision dot data sets prebuilt functions.
21:08:49
So let's do some more visualization in the next video. I'll see you there. Welcome back. In the last video, we used data sets dot image folder to turn all of our image data into tenses. And we did that with the help of our data transform, which is a little pipeline up here to take in some data, or specifically an image, resize it to a value that we've set in our k6464 randomly flip it along the horizontal. We don't necessarily need this, but I've just put
21:09:24
that in there to indicate what happens when you pass an image through a transforms pipeline. And then most importantly, we've turned our images into a torch tensor. So that means that our data, our custom data set, this is so exciting, is now compatible to be used with a pytorch model. So let's keep pushing forward. We're not finished yet. We're going to visualize some samples from the train data data set. So let's, how can we do this? Let's get, we can index on the train data
21:09:56
data set to get a single image and a label. So if we go, can we do train data zero? What does that give us? Okay, so this is going to give us an image tensor. And it's associated label. In this case, it's an image of pizza, because why it's associated label is pizza. So let's take the zero zero. So this is going to be our image. And the label is going to be train data zero. And we're just going to get the first index item there, which is going to be one. And then if we have a look
21:10:35
at them separately, image and label, beautiful. So now one of our target images is in tensor format, exactly how we want it. And it's label is in numeric format as well, which is also exactly how we want it. And then if we wanted to convert this back to a non label, we can go class names
21:10:57
and index on that. And we see pizza. And I mean, non label is in non numeric, we can get it back to string format, which is human understandable. We can just index on class names. So let's print out some information about what's going on here. Print F, we're going to go image tensor. I love F strings if you haven't noticed yet. Image tensor. And we're going to set in new line, we're going to pass it in our image, which is just the image that we've got here.
21:11:26
Then we'll print in some more information about that. This is still all becoming one with the data right where we're slowly finding out information about our data set so that if errors arise later on, we can go, hmm, our image or we're getting a shape error. And I know our images are of this shape or we're getting a data type error, which is why I've got the dot D type here. And that might be why we're getting a data type issue. So let's do one more with the image label,
21:11:53
label, oh, well, actually, we'll do one more. We'll do print, we'll get the label data type as well. Label, this will be important to take note of later on. Type, as I said, three big issues. Shape mismatch, device mismatch, and data type mismatch. Can we get the type of our label? Beautiful. So we've got our image tensor and we've got its shape. It's of torch size 36464. That's exactly how we want it. The data type is torch float 32, which is the default data type
21:12:31
in PyTorch. Our image label is zero and the label data type is of integer. So let's try and plot this and see what it looks like, hey, using matplotlib. So first of all, what do we have to do? Well, we have to rearrange the order of dimensions. In other words, matplotlib likes color channels last. So let's see what looks this looks like. We'll go image per mute. We've done this before, image.permute 120 means we're reordering the dimensions. Zero would usually be here,
21:13:06
except that we've taken the zero dimension, the color channels and put it on the end and shuffled the other two forward. So let's now print out different shapes. I love printing out the change in shapes. It helps me really understand what's going on. Because sometimes I look at a line like this and it doesn't really help me. But if I print out something of what the shapes were originally and what they changed to, well, hey, that's a big help. That's what
21:13:31
Jupiter notebooks are all about, right? So this is going to be color channels first, height, width. And depending on what data you're using, if you're not using images, if you're using text, still knowing the shape of your data is a very good thing. We're going to go image per mute.shape and this should be everything going right is height with color channels on the end here. And we're just going to plot the image. You can never get enough plotting practice.
21:14:04
Plot the image. You're going to go PLT dot figure, we'll pass in fig size equals 10, 7. And then we're going to PLT dot in show. We'll pass in the permuted image, image underscore permutes, and then we'll turn off the axes. And we will set the title to be class names. And we're going to index on the label, just as we did before. And we're going to set the font size equal to 14. So it's nice and big. Here we go. Beautiful. There is our image of pizza.
21:14:41
It is very pixelated because we're going from about 512 as the original size 512 by 512 to 64, 64. I would encourage you to try this out. Potentially, you could use a different image here. So we've indexed on sample zero. Maybe you want to change this to just be a random image and go through these steps here. And then if you'd like to see different transforms, I'd also encourage you to try changing this out, our transform pipeline here, maybe increase the size and see what it looks
21:15:10
like. And if you're feeling really adventurous, you can go into torch vision and look at the transforms library here and then try one of these and see what it does to our images. But we're going to keep pushing forward. We are going to look at another way. Or actually, I think for completeness, let's now turn, we've got a data set. We want to, we wrote up here before that we wanted to turn our images into a data set, and then subsequently a torch utils data
21:15:43
data loader. So we've done this before, by batching our images, or batching our data that we've been working with. So I'd encourage you to give this a shot yourself. Try to go through the next video and create a train data loader using our train data, wherever that is train data, and a test data loader using our test data. So give that a shot and we'll do it together in the next video. We'll turn our data sets into data loaders. Welcome back. How'd you go? In the last
21:16:19
video, I issued you the challenge to turn our data sets into data loaders. So let's do that together now. I hope you gave it a shot. That's the best way to practice. So turn loaded images into data loaders. So we're still adhering to our PyTorch workflow here. We've got a custom data set. We found a way to turn it into tenses in the form of data sets. And now we're going to turn it into a data loader. So we can turn our data sets into iterables or batchify our data.
21:16:50
So let's write down here, a data loader is going to help us turn our data sets into iterables. And we can customize the batch size, write this down. So our model can see batch size images at a time. So this is very important. As we touched on in the last section computer vision, we create a batch size because if we had 100,000 images, chances are if they were all in one data set, there's 100,000 images in the food 101 data set. We're only working with about 200.
21:17:31
If we try to load all 100,000 in one hit, chances are our hardware may run out of memory. And so that's why we matchify our images. So if we have a look at this, NVIDIA SMI, our GPU only has 16 gigabytes. I'm using a Tesla T4 right now, well, has about 15 gigabytes of memory. So if we tried to load 100,000 images into that whilst also computing on them with a PyTorch model, potentially we're going to run out of memory and run into issues. So instead, we can turn them
21:18:03
into a data loader so that our model looks at 32 images at a time and can leverage all of the memory that it has rather than running out of memory. So let's turn our train and test data sets into data loaders, turn train and test data sets into data loaders. Now, this is not just for image data. This is for all kinds of data in PyTorch. Images, text, audio, you name it. So import data loader, then we're going to create a train data loader. We're going to set it equal to data loader.
21:18:42
We're going to pass in a data set. So let's set this to train data. Let's set the batch size. What should we set the batch size to? I'm going to come up here and set a laser capital variable. I'm going to use 32 because 32 is a good batch size. So we'll go 32 or actually, let's start small. Let's just start with a batch size of one and see what happens. Batch size one, number of workers. So this parameter is going to be, this is an important one. I'm going
21:19:11
to, I potentially have covered it before, but I'm going to introduce it again. Is this going to be how many cores or how many CPU cores that is used to load your data? So the higher the better usually and you can set this via OS CPU count, which will count how many CPUs your compute hardware has. So I'll just show you how this works. Import OS and this is a Python OS module. We can do CPU count to find out how many CPUs our Google Colab instance has. Mine has two,
21:19:45
your number may vary, but I believe most Colab instances have two CPUs. If you're running this on your local machine, you may have more. If you're running it on dedicated deep learning hardware, you may even have even more, right? So generally, if you set this to one, it will use one CPU core, but if you set it to OS dot CPU count, it will use as many as possible. So we're just going to leave this as one right now. You can customize this to however you want. And I'm going to shuffle
21:20:16
the training data because I don't want my model to recognize any order in the training data. So I'm going to mix it up. And then I'm going to create the test data loader. Data set equals test data.
21:20:29
And batch size equals one, num workers, I'm going to set this to equal one as well. Again, you can customize each of these, their hyper parameters to whatever you want. Number of workers generally the more the better. And then I'm going to set shuffle equals false for the test data so that if we want to evaluate our models later on, our test data set is always in the same order. So now let's have a look at train data loader, see what happens. And test data loader.
21:21:03
Wonderful. So we get two instances of torch utils dot data dot data loader. And now we can see if we can visualize something from the train data loader, as well as the test data loader. I actually maybe we just visualize something from one of them. So we're not just double handling everything. We get a length here. Wonderful. Because we're using a batch size of one, our lengths of our data loaders are the same as our data sets. Now, of course, this would change
21:21:34
if we set, oh, we didn't even set this to the batch size parameter batch size. Let's come down here and do the same here batch size. So we'll watch this change. If we wanted to look at 32 images at a time, we definitely could do that. So now we have eight batches, because 22, 225 divided by 32 equals roughly eight. And then 75 divided by 32 also equals roughly three. And remember, these numbers are going to be rounded if there are some overlaps. So let's get rid of,
21:22:08
we'll change this back to one. And we'll keep that there. We'll get rid of these two. And let's see what it looks like to plot an image from our data loader. Or at least have a look at it.
21:22:22
Check out the shapes. That's probably the most important point at this time. We've already plotted in our things. So let's iterate through our train data loader. And we'll grab the next one. We'll grab the image and the label. And we're going to print out here. So batch size will now be one. You can change the batch size if you like. This is just again, another way of getting familiar with the shapes of our data. So image shape. Let's go image dot shape. And we're going to
21:22:57
write down here. This shape is going to be batch size. This is what our data loader is going to add to our images is going to add a batch dimension, color channels, height, width. And then print. Let's check out that label shape. Same thing with the labels. It's going to add a batch dimension. Label. And let's see what happens. Oh, we forgot the end of the bracket. Beautiful. So we've got image shape. Our label shape is only one because we have a batch size of one.
21:23:34
And so now we've got batch size one, color channels three, height, width. And if we change this to 32, what do you think's going to happen? We get a batch size of 32, still three color channels, still 64, still 64. And now we have 32 labels. So that means within each batch, we have 32 images. And we have 32 labels. We could use this with a model. I'm going to change this back to one. And I think we've covered enough in terms of loading our data sets. How cool is this?
21:24:11
We've come a long way. We've downloaded a custom data set. We've loaded it into a data set using image folder turned it into tenses using our data transform and now batchified our custom data set in data loaders. We've used these with models before. So if you wanted to, you could go right ahead and build a convolutional neural network to try and find patterns in our image tenses. But in the next video, let's pretend we didn't have this data loader,
21:24:41
this image folder class available to us. How could we load our image data set so that it's compatible? Like our image data set here, how could we replicate this image folder class? So that we could use it with a data loader. Because data load is part of torch utils.data, you're going to see these everywhere. Let's pretend we didn't have the torch vision.data sets image folder helper function. And we'll see in the next video, how we can replicate that functionality.
21:25:15
I'll see you there. Welcome back. So over the past few videos, we've been working out how to get how to get our data from our data folder, pizza, steak, and sushi. We've got images of different food data here. And we're trying to get it into Tensor format. So we've seen how to do that with an existing data loader helper function or data set function in image folder. However, what if image folder didn't exist? And we need to write our own custom data loading function.
21:25:50
Now the premise of this is although it does exist, it's going to be good practice because you might come across a case where you're trying to use a data set where a prebuilt function doesn't exist. So let's replicate the functionality of image folder by creating our own data loading class. So we want a few things. We want to be able to get the class names as a list from our loaded data. And we want to be able to get our class names as a dictionary as well. So the whole goal of this
21:26:22
video is to start writing a function or a class that's capable of loading data from here into Tensor format, capable of being used with the PyTorch's data loader class, like we've done here. So we want to create a data set. Let's start it off. We're going to create another heading here. This is going to be number five, option two, loading image data with a custom data set. So we want a few
21:26:53
functionality steps here. Number one is one, two, be able to load images from file to one, two, be able to get class names from the data set, and three, one, two, be able to get classes as dictionary from the data set. And so let's briefly discuss the pros and cons of creating your own custom data set. We saw option one was to use a pre-existing data set loader helping function from torch vision. And it's going to be quite similar if we go torch vision data sets.
21:27:39
Quite similar if you're using other domain libraries here, there we're going to be data loading utilities. But at the base level of PyTorch is torchutils.data.dataset. Now this is the base data set class. So we want to build on top of this to create our own image folder loading class. So what are the pros and cons of creating your own custom data set? Well, let's discuss some pros. So one pro would be you can create a data set out of almost anything as long as you write
21:28:16
the right code to load it in. And another pro is that you're not limited to PyTorch pre-built data set functions. A couple of cons would be that even though this is to point number one.
21:28:35
So even though you could create a data set out of almost anything, it doesn't mean that it will automatically work. It will work. And of course, you can verify this through extensive testing, seeing if your model actually works, if it actually loads data in the way that you want it. And another con is that using a custom data set requires us to write more code. So often results in us writing more code, which could be prone to errors or performance issues. So typically if
21:29:14
something makes it into the PyTorch standard library or the PyTorch domain libraries,
21:29:22
if functionality makes it into here, it's generally been tested many, many times. And it can kind of be verified that it works quite well with, or if you do use it, it works quite well. Whereas if we write our own code, sure, we can test it ourselves, but it hasn't got the robustness to begin with, that is, we could fix it over time, as something that's included in say the PyTorch standard library. Nonetheless, it's important to be aware of how we could create such a custom data set.
21:29:50
So let's import a few things that we're going to use. We'll import OS, because we're going to be working with Python's file system over here. We're going to import path lib, because we're going to be working with file paths. We'll import torch, we don't need to again, but I'm just doing this for completeness. We're going to import image from pill, the image class, because we want to be opening images. I'm going to import from torch utils dot data. I'm going to import data set,
21:30:20
which is the base data set. And as I said over here, we can go to data sets, click on torch utils data dot data set. This is an abstract class representing a data set. And you'll find that this data set links to itself. So this is the base data set class. Many of the data sets in PyTorch, the prebuilt functions, subclass this. So this is what we're going to be doing it. And as a few notes here, all subclasses should overwrite get item. And you should optionally
21:30:49
overwrite land. These two methods, we're going to see this in a future video. For now, we're just we're just setting the scene here. So from torch vision, we're going to import transforms, because we want to not only import our images, but we want to transform them into tenses. And from the Python's typing module, I'm going to import tuple dict and list. So we can put type hints when we create our class and loading functions. Wonderful. So this is our instance of torch vision
21:31:25
dot data sets image folder, torch vision dot data sets dot image folder. Let's have a look at the train data. So we want to write a function that can replicate getting the classes from a particular directory, and also turning them into an index or dictionary that is. So let's build a helper function to replicate this functionality here. In other words, I'd like to write a helper function that if we pass it in a file path, such as pizza steak sushi or this data folder,
21:31:58
it's going to go in here. And it's going to return the class names as a list. And it's also going to turn them into a dictionary, because it's going to be helpful for later on when we'd like to access the classes and the class to ID X. And if we really want to completely recreate image folder, well, image folder has this functionality. So we'd like that too. So this is just a little high level overview of what we're going to be doing. I might link in here that we're going to subclass this.
21:32:29
So all custom data sets in pie torch, often subclass this. So here's what we're going to be doing. Over the next few videos, we want to be able to load images from a file. Now you could replace images with whatever data that you're working with the same premise will be here. You want to be able to get the class names from the data set and want to be able to get classes as a dictionary from the data set. So we're going to map our samples, our image samples to that class name
21:33:00
by just passing a file path to a function that we're about to write. And some pros and cons of creating a custom data set. We've been through that. Let's in the next video, start coding up a helper function to retrieve these two things from our target directory. In the last video, we discussed the exciting concept of creating a custom data set. And we wrote down a few things that we want to get. We discussed some pros and cons. And we learned that many custom data sets
21:33:34
inherit from torch dot utils dot data data set. So that's what we'll be doing later on. In this video, let's focus on writing a helper function to recreate this functionality. So I'm going to title this 5.1, creating a helper function to get class names. I'm going to turn this into markdown. And if I go into here, so we want to function to let's write down some steps and then we'll code it out. So we'll get the class names, we're going to use OS dot scanner. So it's going
21:34:09
to scanner directory to traverse a target directory. And ideally, the directory is in standard image classification format. So just like the image folder class, our custom data class is going to require our data already be formatted. In the standard image classification format, such as train and test for training and test images, and then images for a particular class are in a particular directory. So let's keep going. And number two, what else do we want it to do? We want
21:34:49
it to raise an error if the class names aren't found. So if this happens, there might be, we want this to enter the fact that there might be something wrong with the directory structure.
21:35:04
And number three, we also want to turn the class names into our dict and a list and return them.
21:35:15
Beautiful. So let's get started. Let's set up the path directory for the target directory. So our target directory is going to be what the directory we want to load directory, if I could spell, we want to load our data from, let's start with the training der, just for an example. So target directory, what do we get? So we're just going to use the training folder as an example to begin with. And we'll go print target der, we'll put in the target
21:35:49
directory, just want to exemplify what we're doing. And then we're going to get the class names from the target directory. So I'll show you the functionality of our scanner. Of course, you could look this up in the Python documentation. So class names found, let's set this to be sorted. And then we'll get the entry name, entry dot name for entry in list. So we're going to get OS list scanner of the image path slash target directory. Let's see what happens when we do this.
21:36:32
Target directory have we got the right brackets here. Now, is this going to work? Let's find out. Oh, image path slash target directory.
21:36:48
What do we get wrong? Oh, we don't need the image path there. Let's put, let's just put target directory there. There we go. Beautiful. So we set up our target directory as been the training to. And so if we just go, let's just do list. What happens if we just run this function here?
21:37:09
Oh, a scanner. Yeah, so there we go. So we have three directory entries. So this is where we're getting entry dot name for everything in the training directory. So if we look in the training directory, what do we have train? And we have one entry for pizza, one entry for sushi, one entry for steak. Wonderful. So now we have a way to get a list of class names. And we could quite easily turn this into a dictionary, couldn't we? Which is exactly what we want to do. We want to recreate
21:37:38
this, which we've done. And we want to recreate this, which is also done. So now let's take this functionality here. And let's turn that into a function. All right, what can we do? What do we call this? I'm going to call this def fine classes. And I'm going to say that it takes in a directory which is a string. And it's going to return. This is where I imported typing from Python type and imported tuple. And I'm going to return a list, which is a list of strings and a dictionary,
21:38:13
which is strings map to integers. Beautiful. So let's keep going. We want to, we want this function to return given a target directory, we want it to return these two things. So we've seen how we can get a list of the directories in a target directory by using OS scanner. So let's write finds the classes are the class folder names in a target directory. Beautiful. And we know that it's going to return a list and a dictionary. So let's do step number one, we want to get the
21:38:54
class names by scanning the target directory. We'll go classes, just we're going to replicate the functionality we've done about, but for any given directory here. So classes equals sorted entry dot name for entry in OS scanner. And we're going to pass at the target directory. If entry dot is dirt, we're just going to make sure it's a directory as well. And so if we just return classes and see what happens. So find classes, let's pass it in our target directory, which is our training directory.
21:39:39
What do we get? Beautiful. So we need to also return class to ID X. So let's keep going. So number two is let's go raise an error. If class names could not be found. So if not classes, let's say raise file, we're going to raise a file not found error. And then let's just write in here F couldn't find any classes in directory. So we're just writing some error checking code here.
21:40:19
So if we can't find a class list within our target directory, we're going to raise this error and say couldn't find any classes in directory, please check file structure. And there's another checkup here that's going to help us as well to check if the entry is a directory. So finally, let's do number three. What do we want to do? So we want to create a dictionary of index labels. So computers, why do we do this? Well, computers prefer numbers rather than strings as labels. So we
21:40:55
can do this, we've already got a list of classes. So let's just create class to ID X equals class name, I for I class name in enumerate classes. Let's see what this looks like.
21:41:18
So we go class names, and then class to ID X, or we can just return it actually. Do we spell enumerate role? Yes, we did. So what this is going to do is going to map a class name to an integer or to I for I class name in enumerate classes. So it's going to go through this, and it's going to go for I. So the first one zero is going to be pizza. Ideally, one will be steak, two will be sushi. Let's see how this goes. Beautiful. Look at that. We've just replicated
21:41:49
the functionality of image folder. So now we can use this helper function in our own custom data set, find classes to traverse through a target directory, such as train, we could do the same for test if we wanted to to. And that way, we've got a list of classes. And we've also got a dictionary mapping those classes to integers. So now let's in the next video move towards sub classing torch utils dot data dot data set. And we're going to fully replicate image folder. So I'll see you there.
21:42:30
In the last video, we wrote a great helper function called find classes that takes in a target directory and returns a list of classes and a dictionary mapping those class names to an integer. So let's move forward. And this time, we're going to create a custom data set. To replicate image folder. Now we don't necessarily have to do this, right, because image folder already exists. And if something already exists in the pie torch library, chances are it's going to be tested well,
21:43:01
it's going to work efficiently. And we should use it if we can. But if we needed some custom functionality, we can always build up our own custom data set by sub classing torch dot utils dot data data set. Or if a pre built data set function didn't exist, well, we're probably going to want to subclass torch utils data dot data set anyway. And if we go into the documentation here, there's a few things that we need to keep in mind when we're creating our own custom data set.
21:43:30
All data sets that represent a map from keys to data samples. So that's what we want to do. We want to map keys, in other words, targets or labels to data samples, which in our case are food images. So we should subclass this class here. Now to note, all subclasses should overwrite get item. So get item is a method in Python, which is going to get an item or get a sample, supporting fetching a data sample for a given key. So for example, if we wanted to get sample
21:44:03
number 100, this is what get item should support and should return us sample number 100. And subclasses could also optionally override land, which is the length of a data set. So return the size of the data set by many sampler implementations and the default options of data loader, because we want to use this custom data set with data loader later on. So we should keep this in mind when we're building our own custom subclasses of torch utils data data set. Let's see
21:44:34
this hands on, we're going to break it down. It's going to be a fair bit of code, but that's all right. Nothing that we can't handle. So to create our own custom data set, we want to number one, first things first is we're going to subclass subclass torch dot utils dot data dot data set. Two, what do we want to do? We want to init our subclass with target directory. So the directory we'd like to get data from, as well as a transform, if we'd like to transform our data. So just like
21:45:18
when we used image folder, we could pass a transform to our data set, so that we could transform the data that we were loading. We want to do the same thing. And we want to create several attributes.
21:45:33
Let's write them down here. We want paths, which will be the parts of our images. What else do we want? We want transform, which will be the transform we'd like to use. We want classes, which is going to be a list of the target classes. And we want class to ID X, which is going to be a dict of the target classes, mapped to integer labels. Now, of course, these attributes will differ depending on your data set. But we're replicating image folder here. So these are just
21:46:16
some of the things that we've seen that come with image folder. But regardless of what data set you're working with, there are probably some things that you want to cross them universal. You probably want all the paths of where your data is coming from, the transforms you'd like to perform on your data, what classes you're working with, and a map of those classes to an index. So let's keep pushing forward. We want to create a function to load images, because after all,
21:46:44
we want to open some images. So this function will open an image. Number five, we want to overwrite the LAN method to return the length of our data set. So just like it said in the documentation,
21:47:05
if you subclass using torch.utils.data, the data set, you should overwrite get item, and you should optionally overwrite LAN. So we're going to, instead of optionally, we are going to overwrite length. And number six, we want to overwrite the get item method to return a given sample when passed an index. Excellent. So we've got a fair few steps here. But if they don't make sense now, it's okay. Let's code it out. Remember our motto, if and doubt, code it out. And if
21:47:45
and doubt, run the code. So we're going to write a custom data set. This is so exciting, because
21:47:51
when you work with prebuilt data sets, it's pretty cool in machine learning. But when you can write code to create your own data sets, and that's, well, that's magic. So number one is we're going to, or number zero is we're going to import torch utils data set, we don't have to rewrite this, we've already imported it, but we're going to do it anyway for completeness. Now step number one is to subclass it subclass torch utils data, the data set. So just like when we built a model,
21:48:25
we're going to subclass and in module, but in this time, we're going to call us our class image folder custom. And we're going to inherit from data set. This means that all the functionality that's contained within torch utils data data set, we're going to get for our own custom class.
21:48:44
Number two, let's initialize. So we're going to initialize
21:48:49
our custom data set. And there's a few things that we'd like, and into our subclass with the target directory, the directory we'd like to get data from, as well as the transform if we'd like to transform our data. So let's write a knit function, a knit, and we're going to go self, target, and target is going to be a string. And we're going to set a transform here, we'll set it equal to none. Beautiful. So this way we can pass in a target directory of images
21:49:24
that we'd like to load. And we can also pass in a transform, just similar to the transforms that we've created previously. So now we're up to number three, which is create several attributes. So let's see what this looks like, create class attributes. So we'll get all of the image paths. So we can do this just like we've done before, self paths equals list, path lib dot path, because what's our target directory going to be? Well, I'll give you a spoiler alert,
21:49:56
it's going to be a path like the test directory, or it's going to be the train directory. Because we're going to use this once for our test directory and our train directory, just like we use the original image folder. So we're going to go through the target directory and find out all of the paths. So this is getting all of the image paths that support or that follow the file name convention of star star dot jpg. So if we have a look at this,
21:50:31
we passed in the test folder. So test is the folder star would mean any of these 123 pizza steak sushi, that's the first star, then slash would go into the pizza directory. The star here would mean any of the file combinations here that end in dot jpg. So this is getting us a list of all of the image paths within a target directory. In other words, within the test directory and within the train directory, when we call these two separately. So let's keep going, we've got all
21:51:02
of the image parts, what else did we have to do? We want to create transforms. So let's set up transforms, self dot transforms equals transform. Oh, we'll just call that transform actually, set up transform equals transform. So we're going to get this from here. And I put it as none because it transform can be optional. So let's create classes and class to ID X attributes, which is the next one on our list, which is here classes and class to ID X. Now, lucky us,
21:51:39
in the previous video, we created a function to return just those things. So let's go self dot classes and self dot class to ID X equals find classes. And we're going to pass in the target der or the target der from here. Now, what's next? We've done step number three, we need
21:52:05
number four is create a function to load images. All right, let's see what this looks like. So number four, create a function to load images. So let's call it load image. And we're going to pass in self. And we'll also pass in an index. So the index of the image we'd like to load. And this is going to return an image dot image. So where does that come from? Well, previously, we imported from pill. So we're going to use Python image library or pillow to import our
21:52:39
images. So we're going to give on a file path from here, such as pizza, we're going to import it with the image class. And we can do that using, I believe it's image dot open. So let's give that a try. I'll just write a note in here, opens an image via a path and returns it. So let's write image path equals self. This is why we got all of the image paths above. So self dot paths. And we're going to index it on the index. Beautiful. And then let's return image dot open image path.
21:53:17
So we're going to get a particular image path. And then we're just going to open it.
21:53:22
So now we're up to step number five, override the land method to return the length of our data set.
21:53:29
This is optional, but we're going to do it anyway. So overwrite. Len. So this just wants to return how many samples we have in our data set. So let's write that def, Len. So if we call Len on our data set instance, it's going to return just how many numbers there are. So let's write this down. Returns the total number of samples. And this is just going to be simply return length or Len of self dot paths. So for our target directory, if it was the training
21:54:08
directory, we'd return the number of image paths that this code has found out here. And same for the test directory. So next, I'm going to go number six is we want to overwrite, we put this up here, the get item method. So this is required if we want to subclass torch utils data data set. So this is in the documentation here. All subclasses should override get item. So we want get item to, if we pass it an index to our data set, we want it to return that particular item. So let's see
21:54:43
what this looks like. Override the get item method to return our particular sample. And now this method is going to leverage get item, all of the code that we've created above. So this is going to go take in self, which is the class itself. And it's going to take in an index, which will be of an integer. And it's going to return a tuple of torch dot tensor and an integer, which is the same thing that gets returned when we index on our training data. So if we have a
21:55:21
look image label equals train data, zero, get item is going to replicate this. We pass it an index here.
21:55:33
Let's check out the image and the label. This is what we have to replicate. So remember train data was created with image folder from torch vision dot data sets. And so we will now get item to return an image and a label, which is a tuple of a torch tensor, where the image is of a tensor here. And the label is of an integer, which is the label here, the particular index as to which this image relates to. So let's keep pushing forward. I'm going to write down here, returns one sample
21:56:08
of data, data and label, X and, or we'll just go XY. So we know that it's a tuple. Beautiful. So let's set up the image. What do we want the image to be? Well, this is where we're going to call on our self dot load image function, which is what we've created up here. Do you see the customization capabilities of creating your own class? So we've got a fair bit of code here, right? But essentially, all we're doing is we're just creating functions that is going to help us
21:56:44
load our images into some way, shape or form. Now, again, I can't stress this enough, regardless of the data that you're working on, the pattern here will be quite similar. You'll just have to change the different functions you use to load your data. So let's load an image of a particular index. So if we pass in an index here, it's going to load in that image. Then what do we do? Well, we want to get the class name, which is going to be self dot paths. And we'll get the index here,
21:57:15
and we can go parent dot name. So this expects path in format data,
21:57:24
folder slash class name slash image dot JPG. That's just something to be aware of. And the class ID X is going to be self dot class to ID X. And we will get the class name here.
21:57:42
So now we have an image by loading in the image here. We have a class name by because our data is going to be or our data is currently in standard image classification format. You may have to change this depending on the format your data is in, we can get the class name from that, and we can get the class ID X by indexing on our attribute up here, our dictionary of class names to indexes. Now we have one small little step. This is transform if necessary. So remember our
21:58:16
transform parameter up here. If we want to transform our target image, well, let's put in if self dot transform if the transform exists, let's pass the image through that transform, transform image and then we're going to also return the class ID X. So do you notice how we've returned a tuple here? This is going to be a torch tensor. If our transform exists and the class ID X is also going to be returned, which is what we want here, X and Y, which is what gets returned here,
21:58:51
image as a tensor label as an integer. So return data label X, Y, and then if the transform doesn't exist, let's just return image class ID X, return untransformed image and label. Beautiful. So that is a fair bit of code there. So you can see the pro of subclassing torch utils data that data set is that we can customize this in almost any way we wanted to to load whatever data that we're working with, well, almost any data. However, because we've written so much code, this may be
21:59:34
prone to errors, which we're going to find out in the next video to see if it actually works. But essentially, all we've done is we've followed the documentation here torch dot utils data dot data set to replicate the functionality of an existing data loader function, namely image folder. So if we scroll back up, ideally, if we've done it right, we should be able to write code like this,
21:59:58
passing in a root directory, such as a training directory, a particular data transform. And we should get very similar instances as image folder, but using our own custom data set class. So let's try that out in the next video. So now we've got a custom image folder class that replicates the functionality of the original image folder, data loader class, or data set class, that is, let's test it out. Let's see if it works on our own custom data.
22:00:32
So we're going to create a transform here so that we can transform our images raw jpeg images into tenses, because that's the whole goal of importing data into pytorch. So let's set up a train transforms compose. We're going to set it to equal to transforms dot compose. And I'm going to pass in a list here, that it's going to be transforms, we're going to resize it to 6464. Whatever the image size will reduce it down to 6464. Then we're going to go transforms dot random horizontal flip. We don't
22:01:12
need to necessarily flip them, but we're going to do it anyway, just to see if it works. And then let's put in here transforms dot to tensor, because our images are getting opened as a pill image, using image dot open. But now we're using the to transform transform from pytorch or torch visions dot transforms. So I'll just put this here. From torch vision dot transforms, that way you know where importing transforms there. And let's create one for the test data set as well, test
22:01:47
transforms, we'll set this up. Oh, excuse me, I need to just go import transforms. And let's go transforms dot compose. And we'll pass in another list, we're going to do the exact same as above, we'll set up resize, and we'll set the size equal to 6464. And then transforms, we're going to go dot to tensor, we're going to skip the data augmentation for test data. Because typically, you don't manipulate your test data in terms of data augmentation, you just convert it into a
22:02:23
tensor, rather than manipulate its orientation, shape, size, etc, etc. So let's run this.
22:02:31
And now let's see how image folder custom class works. Test out image folder custom. Let's go, we'll set up the train data custom is equal to image folder custom. And then we'll set up the target, which is equal to the training directory. And then we'll pass in the transform, which is equal to the train transforms, which we just created above train transforms. And then we're going to, I think that's all we need, actually, we only had two parameters that we're not going
22:03:09
to use a target transform, because our labels, we've got to help a function to transform our labels. So test data custom is going to be image folder custom. And I'm going to set up the target to be equal to the test directory. And the transform is going to be the test transforms from the cell above there. And what's co lab telling me there? Oh, I'm going to set that up. Did we spell something? Oh, we spelled it wrong train transforms. There we go. Beautiful. Now let's have a look at
22:03:40
our train data and test data custom. See if it worked. What do we have? Or we have an image folder custom. Well, it doesn't give us as much rich information as just checking it out as it does for the train data. But that's okay. We can still inspect these. So this is our original one made with image folder. And we've got now train data custom and test data custom. Let's see if we can get some information from there. So let's check the original length of the train data and see if
22:04:13
we can use the land method on our train data custom. Did that work? Wonderful. Now how about we do it for the original test data made with image folder and our custom version made with test data or image folder custom. Beautiful. That's exactly what we want. And now let's have a look at the train data custom. Let's see if the classes attribute comes up. Dot classes. And we'll just leave that there. We'll do the class dot ID X. Yes, it is. So this attribute here is I wonder if we get
22:04:49
information from Google co lab loading. What do we get? Oh, classes to ID X classes load image paths transform. So if we go back up here, all these attributes are from here paths transform classes class to ID X as well as load image. So this is all coming from the code that we wrote our custom data set class. So let's keep pushing forward. Let's have a look at the class to ID X. Do we get the same as what we wanted before? Yes, we do beautiful a dictionary containing our
22:05:21
string names and the integer associations. So let's now check for equality. We can do this by going
22:05:31
check for equality between original image folder data set and image folder custom data set. Now we've kind of already done that here, but let's just try it out. Let's go print. Let's go train data custom dot classes. Is that equal to train? Oh, I don't want three equals train data. The original one classes and also print. Let's do test data custom dot classes. Is this equal to test data? The original one classes. True and true. Now you could try this out. In fact,
22:06:16
it's a little exercise to try it out to compare the others. But congratulations to us, we have replicated the main functionality of the image folder data set class. And so the takeaways from this is that whatever data you have, PyTorch gives you a base data set class to inherit from. And then you can write a function or a class that somehow interacts with whatever data you're working with. So in our case, we load in an image. And then you, as long as you override the land
22:06:49
method and the get item method and return some sort of values, well, you can create your own data set loading function. How beautiful is that? So that's going to help you work with your own custom data sets in PyTorch. So let's keep pushing forward. We've seen analytically that our custom data set is quite similar to the original PyTorch, torch vision dot data sets image folder data set. But you know what I like to do? I like to visualize things. So let's in
22:07:20
the next video, create a function to display some random images from our trained data custom class. It's time to follow the data explorer's motto of visualize, visualize, visualize. So let's create another section. I'm going to write here a title called create a function to display random images. And sure, we've, we've had a look at the different attributes of our custom data set. We see that it gives back a list of different class names. We see that the lengths are similar
22:07:57
to the original, but there's nothing quite like visualizing some data. So let's go in here. We're going to write a function, a helper function. So step number one, we need to take in a data set. So one of the data sets that we just created, whether it be trained data custom or trained data. And a number of other parameters, such as class names and how many images to visualize. And then step number two is to prevent the display getting out of hand. Let's cap the number of
22:08:39
images to see at 10. Because look, if our data set is going to be thousands of images and we want to put in a number of images to look at, let's just make sure it's the maximum is 10. That should be enough. So we'll set the random seed for reproducibility. Number four is, let's get a list of random samples. So we want random sample indexes, don't just get rid of this s from what do we want it from from the target data set. So we want to take in a data set, and we want to count the number of
22:09:15
images we're seeing, we want to set a random seed. And do you see how much I use randomness here to really get an understanding of our data? I really, really, really love harnessing the power of randomness. So we want to get a random sample of indexes from all of our data set. And then we're going to set up a matplotlib plot. Then we want to loop through the random sample images. And plot them with matplotlib. And then as a side to this one, step seven is we need to make sure
22:09:50
the dimensions of our images line up with matplotlib. So matplotlib needs a height width color channels. All right, let's take it on, hey? So number one is create a function to take in a data set. So we're going to call this def, let's call it def display random images going to be one of our helper functions. We've created a few type of functions like this. But let's take in a data set, which is torch utils of type that is of type data set. Then we're going to take in classes,
22:10:27
which is going to be a list of different strings. So this is going to be our class names for whichever data set we're using. I'm going to set this equal to none. And then we're going to take in n, which is the number of images we'd like to plot. And I'm going to set this to 10 by default. So we can see 10 images at a time, 10 random images, that is, do we want to display the shape? Let's set that equal to true, so that we can display what the shape of the images, because we're passing
22:10:54
it through our transform as it goes into a data set. So we want to see what the shape of our images are just to make sure that that's okay. And we can also let's set up a seed, which is going to be an integer, and we'll set that to none to begin with as well. Okay, so step number two, what do we have above? We have to prevent the display getting out of hand, let's cap the number of images to see at 10. So we've got n is by default, it's going to be 10, but let's just make
22:11:23
sure that it stays there. Adjust display, if n is too high. So if n is greater than 10, let's just readjust this, let's set n equal to 10, and display shape, we'll turn off the display shape, because if we have 10 images, our display may get out of hand. So just print out here for display purposes, and shouldn't be larger than 10, setting to 10, and removing shape display. Now I only know this because I've had experience cooking this dish before.
22:12:01
In other words, I've written this type of code before. So you can customize the beautiful thing about Python and PyTorch, as you can customize these display functions in any way you see fit. So step number three, what are we doing? Set the random seed for reproducibility. Okay, set the seed. So if seed, let's set random dot seed equal to that seed value, and then we can keep and then we can keep going. So number four is let's get some random sample indexes. So we can do
22:12:32
that by going get random sample indexes, which is step number four here. So we've got a target data set that we want to inspect. We want to get some random samples from that. So let's create a random samples IDX list. And I'm going to randomly sample from a length of our data set, or sorry, a range of the length of our data set. And I'll show you what this means in a second.
22:13:00
And the K, excuse me, have we got enough brackets there? I always get confused with the brackets. The K is going to be n. So in this case, I want to randomly sample 10 images from the length of our data set or 10 indexes. So let's just have a look at what this looks like. We'll put in here, our train data custom here. So this is going to take a range of the length of our train data custom, which is what 225. We looked at that before, just up here, length of this. So between zero
22:13:32
and 255, we're going to get 10 indexes if we've done this correctly. Beautiful. So there's 10 random samples from our train data custom, or 10 random indexes, that is. So we're up to step number five, which was loop through the random sample images or indexes. Let's create this to indexes, indexes and plot them with matplotlib. So this is going to give us a list here. So let's go loop through random indexes and plot them with matplotlib. Beautiful. So for
22:14:10
i tug sample in enumerate, let's enumerate through the random, random samples, idx list. And then we're going to go tug image and tug label, because all of the samples in our target data set are in the form of tuples. So we're going to get the target image and the target label, which is going to be data set tug sample. We'll take the index. So it might be one of these values here. We'll index on that. And the zero index will be the image. And then we'll go on the data set as
22:14:51
well. We'll take the tug sample index. And then the index number one will be the label of our target sample. And then number seven, oh, excuse me, we've missed a step. That should be number six.
22:15:08
Did you catch that? Number five is setup plot. So we can do this quite easily by going plot figure. This is so that each time we iterate through another sample, we're going to have quite a big figure here. So we set up the plot outside the loop so that we can add a plot to this original plot here. And now this is number seven, where we make sure the dimensions of our images line up with matplotlib. So if we recall by default, pytorch is going to turn our image dimensions into
22:15:39
what color channels first, however, matplotlib prefers color channels last. So let's go adjust, tensor dimensions for plotting. So let's go tag image. Let's call this tag image adjust equals tag image dot commute. And we're going to alter the order of the indexes. So this is going to go from color channels or the dimensions that is height width. And we're going to change this width, if I could spell, to height width color channels. Beautiful. That one will probably catch you off
22:16:23
guard a few times. But we've seen it a couple of times now. So we're going to keep going with this plot adjusted samples. So now we can add a subplot to our matplotlib plot. And we want to create, we want one row of n images, this will make a lot more sense when we visualize it. And then for the index, we're going to keep track of i plus one. So let's keep going. So then we're going to go plot in show. And I'm going to go tug image adjust. So I'm going to plot this image here. And then
22:17:01
let's turn off the axis. And we can go if the classes variable exists, which is up here, a list of classes, let's adjust the title of the plot to be the particular index in the class list. So title equals f class. And then we're going to put in here classes. And we're going to index on that with the target label index, which is going to come from here. Because that's going to be a new numerical format. And then if display shape, let's set the title equal to title plus f. We're going
22:17:42
to go new line shape. This is going to be the shape of the image, tug image adjust dot shape.
22:17:50
And then we'll set the title to PLT dot title. So you see how if we have display shape, we're just adjusting the title variable that we created here. And then we're putting the title onto the plot. So let's see how this goes. That is quite a beautiful function. Let's pass in one of our data sets and see what it looks like. Let's plot some random images. So which one should we start with first? So let's display random images from the image folder created data sets. So this is the
22:18:23
inbuilt pytorch image folder. Let's go display random images, the function we just created above. We're going to pass in the train data. And then we can pass in the number of images. Let's have a look at five. And the classes is going to be the class names, which is just a list of our different class names. And then we can set the seed, we want it to be random. So we'll just set the seed to equal none. Oh, doesn't that look good? So this is from our original train data
22:18:55
made with image folder. So option number one up here, option one, there we go. And we've passed in the class name. So this is sushi resize to 64, 64, three, same with all of the others, but from different classes. Let's set the seed to 42, see what happens. I get these images, we got a sushi, we got a pizza, we got pizza, sushi pizza. And then if we try a different one, we just go none. We get random images again, wonderful. Now let's write the same code,
22:19:29
but this time using our train data custom data set. So display random images from the image folder custom data set. So this is the one that we created display random images. I'm going to pass in train data custom, our own data set. Oh, this is exciting. Let's set any equal to 10 and just see see how far we can go with with our plot. Or maybe we set it to 20 and just see if our code for adjusting the plot makes sense. Class names and seed equals, I'm going to put in 42 this time.
22:20:08
There we go. For display purposes, and shouldn't be larger than 10 setting to 10 and removing shape display. So we have a stake image, a pizza image, pizza, steak pizza, pizza, pizza, pizza, steak, pizza. If we turn off the random seed, we should get another 10 random images here. Beautiful. Look at that. Steak, steak, sushi, pizza, steak, sushi class. I'm reading out the different things here. Pizza, pizza, pizza, pizza. Okay. So it looks like our custom data set
22:20:41
is working from both a qualitative standpoint, looking at the different images and a quantitative. How about we change it to five and see what it looks like? Do we have a different shape? Yes, we do the same shape as above. Wonderful. Okay. So we've got train data custom. And we've got train data, which is made from image folder. But the premises remain, we've built up a lot of different ideas. And we're looking at things from different points of view. We are
22:21:10
getting our data from the folder structure here into tensor format. So there's still one more step that we have to do. And that's go from data set to data loader. So in the next video, let's see how we can turn our custom loaded images, train data custom, and test data custom into data loaders. So you might want to go ahead and give that a try yourself. We've done it before up here. Turn loaded images into data loaders. We're going to replicate the same thing as we did
22:21:40
in here for our option number two, except this time we'll be using our custom data set. I'll see you in the next video. I'll take some good looking images and even better that they're from our own custom data set. Now we've got one more step. We're going to turn our data set into a data loader. In other words, we're going to batchify all of our images so they can be used with the model. And I gave you the challenge of trying this out yourself in the last video. So I hope
22:22:12
you gave that a go. But let's see what that might look like in here. So I'm going to go 5.4. Let's go. What should we call this? So turn custom loaded images into data loaders. So this is just goes to show that we can write our own custom data set class. And we can still use it with PyTorch's data loader. So let's go from utils torch dot utils that is utils dot data import data loader. We'll get that in here. We don't need to do that again, but I'm just doing it for
22:22:47
completeness. So we're going to set this to train data loader custom. And I'm going to create an instance of data loader here. And then inside I'm going to pass the data set, which is going to be train data custom. I'm just going to set a universal parameter here in capitals for batch size equals 32. Because we can come down here, we can set the batch size, we're going to set this equal to 32. Or in other words, the batch size parameter we set up there, we can set the number of workers
22:23:17
here as well. If you set to zero, let's go see what the default is actually torch utils data loader. What's the default for number of workers? Zero. Okay, beautiful. And recall that number of workers is going to set how many cores load your data with a data loader. And generally higher is better. But you can also experiment with this value and see what value suits your model and your hardware the best. So just keep in mind that number of workers is going to alter how much
22:23:52
compute your hardware that you're running your code on uses to load your data. So by default, it's set to zero. And then we're going to shuffle the training data. Wonderful. And let's do the same for the test data loader. We'll create test data loader custom. And I'm going to create a new instance. So let me make a few code cells here of data loader, and create a data set or pass in the data set parameter as the test data custom. So again, these data sets are what we've created
22:24:24
using our own custom data set class. I'm going to set the batch size equal to batch size. And let's set the number workers equal to zero. In a previous video, we've also set it to CPU count. You can also set it to one. You can hard code it to four all depends on what hardware you're using. I like to use OPA OS dot CPU count. And then we're not going to shuffle the test data.
22:24:56
False. Beautiful. And let's have a look at what we get here. Train data loader custom and test data loader custom. And actually, I'm just going to reset this instead of being OOS CPU count. I'm going to put it back to zero, just so we've got it in line with the one above. And of course, numb workers, we could also set this numb workers equals zero or OS dot CPU count. And then we could come down here and set this as numb workers and numb workers.
22:25:30
And let's have a look to see if it works. Beautiful. So we've got two instances of utils.data.data loader. Now, let's just get a single sample from the train data loader here, just to make sure the image shape and batch size is correct. Get image and label from custom data loader. We want image custom. And I'm going to go label custom equals next. And I'm going to iter over the train data loader custom. And then let's go print out the shapes. We want image custom dot shape and label
22:26:10
custom. Do we get a shape here? Beautiful. There we go. So we have shape here of 32, because that is our batch size. Then we have three color channels, 64, 64, which is in line with what? Which is in line with our transform that we set all the way up here. Transform. We transform our image. You may want to change that to something different depending on the model you're using, depending on how much data you want to be comprised within your image. Recall, generally a larger
22:26:41
image size encodes more information. And this is all coming from our original image folder custom data set class. So look at us go. And I mean, this is a lot of code here or a fair bit of code, right? But you could think of this as like you write it once. And then if your data set continues to be in this format, well, you can use this over and over again. So you might put this, this image folder custom into a helper function file over here, such as data set dot pie or something like
22:27:11
that. And then you could call it in future code instead of rewriting it all the time. And so that's just exactly what pytorch is done with taught vision dot data sets dot image folder. So we've got some shapes here. And if we wanted to change the batch size, what do we do? We just change it like that 64. Remember, a good batch size is also a multiple of eight, because that's going to help out computing. And batch size equals one. We get a batch size equal of one. We've been through a
22:27:43
fair bit. But we've covered a very important thing. And that is loading your own data with a custom data set. So generally, you will be able to load your own data with an existing data loading function or data set function from one of the torch domain libraries, such as torch audio, torch text, torch vision, torch rack. And later on, when it's out of beta, torch data. But if you need to create your own custom one, while you can subclass torch dot utils dot data, dot data set, and then add
22:28:13
your own functionality to it. So let's keep pushing forward. Previously, we touched a little bit on transforming data. And you may have heard me say that torch vision transforms can be used for data augmentation. And if you haven't, that is what the documentation says here. But data augmentation is manipulating our images in some way, shape or form, so that we can artificially increase the diversity of our training data set. So let's have a look at that more in the next video. I'll see you
22:28:46
there. Over the last few videos, we've created functions and classes to load in our own custom data set. And we learned that one of the biggest steps in loading a custom data set is transforming your data, particularly turning your target data into tenses. And we also had a brief look at the torch vision transforms module. And we saw that there's a fair few different ways that we can transform our data. And that one of the ways that we can transform our image data is through
22:29:21
augmentation. And so if we went into the illustration of transforms, let's have a look at all the different ways we can do it. We've got resize going to change the size of the original image. We've got center crop, which will crop. We've got five crop. We've got grayscale. We've got random transforms. We've got Gaussian blur. We've got random rotation, random caffeine, random crop. We could keep going. And in fact, I'd encourage you to check out all of the different options here.
22:29:51
But oh, there's auto augment. Wonderful. There's random augment. This is what I was hinting at. Data augmentation. Do you notice how the original image gets augmented in different ways here? So it gets artificially changed. So it gets rotated a little here. It gets dark and a little here or maybe brightened, depending how you look at it, it gets shifted up here. And then the colors kind of change here. And so this process is known as data augmentation, as we've hinted at.
22:30:21
And we're going to create another section here, which is number six, other forms of transforms. And this is data augmentation. So how could you find out about what data augmentation is? Well, you could go here. What is data augmentation? And I'm sure there's going to be plenty of resources here. Wikipedia. There we go. Data augmentation in data analysis are techniques used to increase the amount of data by adding slightly modified copies of already existing data
22:30:55
or newly created synthetic data from existing data. So I'm going to write down here, data augmentation is the process of artificially adding diversity to your training data. Now, in the case of image data, this may mean applying various image transformations to the training images. And we saw a whole bunch of those in the torch vision transformed package. But now let's have a look at one type of data augmentation in particular. And that is trivial
22:31:35
augment. But just to illustrate this, I've got a slide here ready to go. We've got what is data augmentation. And it's looking at the same image, but from different perspectives. And we do this, as I said, to artificially increase the diversity of a data set. So if we imagine our original images over here on the left, and then if we wanted to rotate it, we could apply a rotation transform. And then if we wanted to shift it on the vertical and the horizontal axis,
22:32:04
we could apply a shift transform. And if we wanted to zoom in on the image, we could apply a zoom transform. And there are many different types of transforms. As I've got a note here, there are many different kinds of data augmentation, such as cropping, replacing, shearing. And this slide only demonstrates a few. But I'd like to highlight another type of data augmentation. And that is one used to recently train pytorch torch vision image models to state
22:32:34
of the art levels. So let's take a look at one particular type of data augmentation,
22:32:43
used to train pytorch vision models to state of the art levels.
22:32:54
Now, just in case you're not sure why we might do this, we would like to increase the diversity of our training data so that our images become harder for our model to learn. Or it gets a chance to view the same image from different perspectives so that when you use your image classification model in practice, it's seen the same sort of images, but from many different angles. So hopefully it learns patterns that are generalizable to those different angles.
22:33:23
So this practice, hopefully, results in a model that's more generalizable to unseen data.
22:33:36
And so if we go to torch vision, state of the art, here we go. So this is a recent blog post by the pytorch team, how to train state of the art models, which is what we want to do, state of the art means best in business, otherwise known as soda. You might see this acronym quite often using torch visions latest primitives. So torch vision is the package that we've been using to work with vision data. And torch vision has a bunch of primitives, which are,
22:34:08
in other words, functions that help us train really good performing models. So blog post here. And if we jump into this blog post and if we scroll down, we've got some improvements here. So there's an original ResNet 50 model. ResNet 50 is a common computer vision architecture. So accuracy at one. So what do we have? Well, let's just say they get a boost in what the previous results were. So if we scroll down, there is a type of data augmentation here. So if we add up
22:34:43
all of the improvements that they used, so there's a whole bunch here. Now, as your extra curriculum, I'd encourage you to look at what the improvements are. You're not going to get them all the first go, but that's all right. Blog posts like this come out all the time and the recipes are continually changing. So even though I'm showing you this now, this may change in the future. So I just scroll down to see if this table showed us what the previous results were. Doesn't look like it does.
22:35:09
Oh, no, there's the baseline. So 76 and with all these little additions, it got right up to nearly 81. So nearly a boost of 5% accuracy. And that's pretty good. So what we're going to have a look at is trivial augment. So there's a bunch of different things such as learning rate optimization, training for longer. So these are ways you can improve your model. Random erasing of image data, label smoothing, you can add that as a parameter to your loss functions, such as cross entropy loss,
22:35:37
mix up and cut mix, weight decay tuning, fixed res mitigations, exponential moving average, which is EMA, inference resize tuning. So there's a whole bunch of different recipe items here, but we're going to focus on what we're going to break it down. Let's have a look at trivial augment. So we'll come in here. Let's look at trivial augment. So if we wanted to look at trivial augment, can we find it in here? Oh, yes, we can. It's right here. Trivial augment.
22:36:06
So as you'll see, if you pass an image into trivial augment, it's going to change it in a few different ways. So if we go into here, let's write that down. So let's see this in action on some of our own data. So we'll import from torch vision, import transforms. And we're going to create a train transform, which is equal to transforms dot compose. We'll pass it in there. And this is going to be very similar to what we've done before in terms of composing a transform. What do we
22:36:45
want to do? Well, let's say we wanted to resize one of our images or an image going through this transform. Let's change its size to 224224, which is a common size in image classification. And then it's going to go through transforms. We're going to pass in trivial augment wide. And there's a parameter here, which is number of magnitude bins, which is basically a number from 0 to 31, 31 being the max of how intense you want the augmentation to happen. So say we, we only put this as
22:37:22
5, our augmentation would be of intensity from 0 to 5. And so in that case, the maximum wouldn't be too intense. So if we put it to 31, it's going to be the max intensity. And what I mean by intensity is say this rotation, if we go on a scale of 0 to 31, this may be a 10, whereas 31 would be completely rotating. And same with all these others, right? So the lower this number, the less the maximum up a bound of the applied transform will be. Then if we go transforms dot to tensor,
22:37:59
wonderful. So there we've just implemented trivial augment. How beautiful is that? That is from the PyTorch torch vision transforms library. We've got trivial augment wide. And it was used trivial augment to train the latest state of the art vision models in the PyTorch torch vision models library or models repository. And if you wanted to look up trivial augment, how could you find that? You could search it. Here is the paper if you'd like to read it. Oh, it's implemented.
22:38:31
It's actually a very, very, I would say, let's just say trivial augment. I didn't want to say simple because I don't want to downplay it. Trivial augment leverages the power of randomness quite beautifully. So I'll let you read more on there. I would rather try it out on our data and visualize it first. Test transform. Let's go transforms compose. And you might have the question of which transforms should I use with my data? Well, that's the million dollar question,
22:38:58
right? That's the same thing as asking, which model should I use for my data? There's a fair few different answers there. And my best answer will be try out a few, see what work for other people like we've done here by finding that trivial augment worked well for the PyTorch team. Try that on your own problems. If it works well, excellent. If it doesn't work well, well, you can always excuse me. We've got a spelling mistake. If it doesn't work well,
22:39:24
well, you can always set up an experiment to try something else. So let's test out our augmentation pipeline. So we'll get all the image paths. We've already done this, but we're going to do it anyway. Again, just to reiterate, we've covered a fair bit here. So I might just rehash on a few things. We're going to get list, image path, which is our, let me just show you our image path. We just want to get all of the images within this file.
22:39:52
So we'll go image path dot glob, glob together all the files and folders that match this pattern. And then if we check, what do we get? We'll check the first 10. Beautiful. And then we can leverage our function from the four to plot some random images, plot random images.
22:40:14
We'll pass in or plot transformed random transformed images. That's what we want. Let's see what it looks like when it goes through our trivial augment. So image paths, equals image part list. This is a function that we've created before, by the way, transform equals train transform, which is the transform we just created above that contains trivial augment.
22:40:40
And then we're going to put n equals three for five images. And we'll do seed equals none to plot. Oh, sorry, n equals three for three images, not five. Beautiful. And we'll set the seed equals none, by the way. So look at this. We've got class pizza. Now trivial augment, it resized this. Now, I'm not quite sure what it did to transform it per se. Maybe it got a little bit darker. This one looks like it's been the colors have been manipulated in some way, shape,
22:41:09
or form. And this one looks like it's been resized and not too much has happened to that one from my perspective. So if we go again, let's have a look at another three images. So trivial augment works. And what I said before, it harnesses the power of randomness. It kind of selects randomly from all of these other augmentation types, and applies them at some level of intensity. So all of these ones here, trivial augment is just going to select summit random, and then
22:41:38
apply them some random intensity from zero to 31, because that's what we've set on our data. And of course, you can read a little bit more in the documentation, or sorry, in the paper here. But I like to see it happening. So this one looks like it's been cut off over here a little bit. This one again, the colors have been changed in some way, shape, or form. This one's been darkened. And so do you see how we're artificially adding diversity to our training data set? So instead
22:42:04
of all of our images being this one perspective like this, we're adding a bunch of different angles and telling our model, hey, you got to try and still learn these patterns, even if they've been manipulated. So we'll try one more of these. So look at that one. That's pretty manipulated there, isn't it? But it's still an image of stake. So that's what we're trying to get our model to do is still recognize this image as an image of stake, even though it's been
22:42:28
manipulated a bit. Now, will this work or not? Hey, it might, it might not, but that's all the nature of experimentation is. So play around. I would encourage you to go in the transforms documentation like we've just done, illustrations, change this one out, trivial augment wine, for another type of augmentation that you can find in here, and see what it does to some of our images randomly. I've just highlighted trivial augment because it's what the PyTorch team have
22:42:57
used in their most recent blog post for their training recipe to train state-of-the-art vision models. So speaking of training models, let's move forward and we've got to build our first model for this section. I'll see you in the next video.
22:43:15
Welcome back. In the last video, we covered how the PyTorch team used trivial augment wide, which is the latest state-of-the-art in data augmentation at the time of recording this video to train their latest state-of-the-art computer vision models that are within torch vision. And we saw how easily we could apply trivial augment thanks to torch vision dot transforms. And we'll just see one more of those in action, just to highlight what's going on.
22:43:45
So it doesn't look like much happened to that image when we augmented, but we see this one has been moved over. We've got some black space there. This one has been rotated a little, and now we've got some black space there. But now's time for us to build our first computer vision model on our own custom data set. So let's get started. We're going to go model zero. We're going to reuse the tiny VGG architecture, which we covered in the computer vision section.
22:44:11
And the first experiment that we're going to do, we're going to build a baseline, which is what we do with model zero. We're going to build it without data augmentation. So rather than use trivial augment, which we've got up here, which is what the PyTorch team used to train their state-of-the-art computer vision models, we're going to start by training our computer vision model without data augmentation. And then so later on, we can try one to see
22:44:36
with data augmentation to see if it helps or doesn't. So let me just put a link in here, CNN explainer. This is the model architecture that we covered in depth in the last section. So we're not going to go spend too much time here. All you have to know is that we're going to have an input of 64, 64, 3 into multiple different layers, such as convolutional layers, relio layers, max pool layers. And then we're going to have some output layer that suits the
22:45:03
number of classes that we have. In this case, there's 10 different classes, but in our case, we have three different classes, one for pizza, steak, and sushi. So let's replicate the tiny VGG architecture from the CNN explainer website. And this is going to be good practice, right? We're not going to spend too much time referencing their architecture. We're going to spend more time coding here. But of course, before we can train a model, what do we have to do? Well,
22:45:35
let's go 7.1. We're going to create some transforms and loading data. We're going to load data for model zero. Now, we could of course use some of the variables that we already have loaded. But we're going to recreate them just to practice. So let's create a simple transform. And what is our whole premise of loading data for model zero? We want to get our data from the data folder, from pizza, steak sushi, from the training and test folders, from their respective folders,
22:46:10
we want to load these images and turn them into tenses. Now we've done this a few times now. And one of the ways that we can do that is by creating a transform equals transforms dot compose. And we're going to pass in, let's resize it. So transforms dot resize, we're going to resize our images to be the same size as the tiny VGG architecture on the CNN explainer website. 64 64 three. And then we're also going to pass in another transform to tensor. So that our
22:46:48
images get resized to 64 64. And then they get converted into tenses. And particularly, these values within that tensor are going to be between zero and one. So there's our transform. Now we're going to load some data. If you want to pause the video here and try to load it yourself, I'd encourage you to try out option one, loading image data using the image folder class, and then turn that data set, that image folder data set into a data loader. So batchify it so
22:47:20
that we can use it with a pytorch model. So give that a shot. Otherwise, let's go ahead and do that together. So one, we're going to load and transform data. We've done this before, but let's just rehash on it what we're doing. So from torch vision import data sets, then we're going to create the train data simple. And I call this simple because we're going to use at first a simple transform, one with no data augmentation. And then later on for another modeling experiment,
22:47:52
we're going to create another transform one with data augmentation. So let's put this here data sets image folder. And let's go the route equals the training directory. And then the transform is going to be what? It's going to be our simple transform that we've got above. And then we can put in test data simple here. And we're going to create data sets dot image folder. And then we're going to pass in the route as the test directory. And we'll pass in the
22:48:22
transform is going to be the simple transform again above. So we're performing the same transformation here on our training data, and on our testing data. Then what's the next step we can do here? Well, we can to turn the data sets into data loaders. So let's try it out. First, we're going to import OS, then from torch dot utils dot data, we're going to import data loader. And then we're going to set up batch size and number of workers. So let's go batch size.
22:48:58
We're going to use a batch size of 32 for our first model.
22:49:03
Numb workers, which will be the number of excuse me, got a typo up here classic number of workers, which will be the what the number of CPU cores that we dedicate towards loading our data. So let's now create the data loaders. We're going to create train data loader simple, which will be equal to data loader. And the data set that goes in here will be train data simple. Then we can set the batch size equal to the batch size parameter that we just created,
22:49:37
or hyper parameter that is, recall a hyper parameter is something that you can set yourself. We would like to shuffle the training data. And we're going to set numb workers equal to numb workers. So in our case, how many calls does Google Colab have? Let's just run this. Find out how many numb workers there are. I think there's going to be two CPUs. Wonderful. And then we're going to do the same thing for the test data loader. Test data loader simple. We're going to go data loader.
22:50:14
We'll pass in the data set here, which is going to be the test data simple. And then we're going to go batch size equals batch size. We're not going to shuffle the test data set. And then the numb workers will just set it to the same thing as we've got above. Beautiful. So I hope you gave that a shot, but now do you see how quickly we can get our data loaded if it's in the right format? I know we spent a lot of time going through all of these steps over multiple videos and
22:50:46
writing lots of code, but this is how quickly we can get set up to load our data. We create a simple transform, and then we load in and transform our data at the same time. And then we turn the data sets into data loaders just like this. Now we're ready to use these data loaders with a model. So speaking of models, how about we build the tiny VGG architecture in the next video? And in fact, we've already done this in notebook number three. So if you want to refer back to the model
22:51:15
that we built there, right down here, which was model number two, if you want to refer back to this section and give it a go yourself, I'd encourage you to do so. Otherwise, we'll build tiny VGG architecture in the next video. Welcome back. In the last video, we got set up starting to get ready to model our first custom data set. And I issued you the challenge to try and replicate the tiny VGG architecture from the CNN explainer website, which we covered in notebook number
22:51:47
three. But now let's see how fast we can do that together. Hey, I'm going to write down here section seven point two. And I know we've already coded this up before, but it's good practice to see what it's like to build pytorch models from scratch, create tiny VGG model class. So the model is going to come from here. Previously, we created our model, there would have been one big change from the model that we created in section number three, which is that our model in section number three
22:52:18
used black and white images. But now the images that we have are going to be color images. So there's going to be three color channels rather than one. And there might be a little bit of a trick that we have to do to find out the shape later on in the classifier layer. But let's get started. We've got class tiny VGG, we're going to inherit from nn.module. This is going to be
22:52:44
the model architecture copying tiny VGG from CNN explainer. And remember that it's a it's quite a common practice in machine learning to find a model that works for a problem similar to yours and then copy it and try it on your own problem. So I only want two underscores there. We're going to initialize our class. We're going to give it an input shape, which will be an int. We're going to say how many hidden units do we want, which will also be an int. And we're going
22:53:17
to have an output shape, which will be an int as well. And it's going to return something none of type none. And if we go down here, we can initialize it with super dot underscore init.
22:53:34
Beautiful. And now let's create the first COM block. So COM block one, which we'll recall will be this section of layers here. So COM block one, let's do an nn.sequential to do so. Now we need com relu com relu max pool. So let's try this out. And then com to D. The in channels is going to be the input shape of our model. The input shape parameter. The out channels is going to be the number of hidden units we have, which is from
22:54:10
Oh, I'm gonna just put enter down here input shape hidden units. We're just getting those to there. Let's set the kernel size to three, which will be how big the convolving window will be over our image data. There's a stride of one and the padding equals one as well. So these are the similar parameters to what the CNN explainer website uses. And we're going to go and then relu. And then we're going to go and then com to D. And I want to stress that even if someone
22:54:43
else uses like certain values for these, you don't have to copy them exactly. So just keep that in mind. You can try out various values of these. These are all hyper parameters that you can set yourself. Hidden units, out channels, equals hidden units as well. Then we're going to go kernel size equals three stride equals one. And we're going to put padding equals one as well. Then we're going to have another relu layer. And I believe I forgot my comma up here.
22:55:15
Another relu layer here. And we're going to finish off
22:55:21
with an N dot max pool 2D. And we're going to put in the kernel size. These equals two and the stride here equals two. Wonderful. So oh, by the way, for max pool 2D, the default stride value is same as the kernel size. So let's have a go here. What can we do now? Well, we could just replicate this block as block two. So how about we copy this down here? We've already had enough practice writing this sort of code. So we're going to
22:56:00
go comp block two, but we need to change the input shape here. The input shape of this block two is going to receive the output shape here. So we need to line those up. This is going to be hidden units. Hidden units. And I believe that's all we need to change there. Beautiful. So let's create the classifier layer. And the classifier layer recall is going to be this output layer here. So we need at some point to add a linear layer. That's going to have a number of outputs
22:56:33
equal to the number of classes that we're working with. And in this case, the number of classes is 10. But in our case, our custom data set, we have three classes, pizza, steak, sushi. So let's create a classifier layer, which will be an end sequential. And then we're going to pass in an end dot flatten to turn the outputs of our convolutional blocks into feature vector into a feature vector site. And then we're going to have an end dot linear. And the end features, do you remember my
22:57:03
trick for calculating the shape in features? I'm going to put hidden units here for the time being. Out features is going to be output shape. So I put hidden units here for the time being because we don't quite yet know what the output shape of all of these operations is going to be. Of course, we could calculate them by hand by looking up the formula for input and output shapes of convolutional layers. So the input and output shapes are here. But I prefer to just do it programmatically and let
22:57:34
the errors tell me where I'm wrong. So we can do that by doing a forward pass. And speaking of a forward pass, let's create a forward method, because every time we have to subclass an end dot module, we have to override the forward method. We've done this a few times. But as you can see, I'm picking up the pace a little bit because you've got this. So let's pass in the conv block one, we're going to go X, then we're going to print out x dot shape. And then we're going to reassign
22:58:03
X to be self.com block two. So we're passing it through our second block of convolutional layers, print X dot shape to check the shape here. Now this is where our model will probably error is because the input shape here isn't going to line up in features, hidden units, because we've passed all of the output of what's going through comp block one, comp block two to a flatten layer, because we want a feature vector to go into our nn.linear layer, our output layer, which has an
22:58:32
out features size of output shape. And then we're going to return X. So I'm going to print x dot shape here. And I just want to let you in on one little secret as well. We haven't covered this before, but we could rewrite this entire forward method, this entire stack of code, by going return self dot classifier, and then going from the outside in. So we could pass in comp block two here, comp block two, and then self comp block one, and then X on the inside.
22:59:03
So that is essentially the exact same thing as what we've done here, except this is going to benefits from operator fusion. Now this topic is beyond the scope of this course, essentially, all you need to know is that operator fusion behind the scenes speeds up how your GPU performs computations. So all of these are going to happen in one step, rather than here, we are reassigning X every time we make a computation through these layers.
22:59:33
So we're spending time going from computation back to memory, computation back to memory, whereas this kind of just chunks it all together in one hit. If you'd like to read more about this, I'd encourage you to look up the blog post, how to make your GPUs go bur from first principles, and bur means fast. That's why I love this post, right? Because it's half satire, half legitimately, like GPU computer science. So if you go in here,
23:00:04
yeah, here's what we want to avoid. We want to avoid all of this transportation between memory and compute. And then if we look in here, we might have operator fusion. There we go. This is operator fusion, the most important optimization in deep learning compilers. So I will link this, making deep learning go bur from first principles by Horace Hare, a great blog post that I really like, right here. So if you'd like to read more on that,
23:00:31
it's also going to be in the extracurricular section of the course. So don't worry, it'll be there. Now, we've got a model. Oh, where do we, where do we forget a comma? Right here, of course we did.
23:00:47
And we've got another, we forgot another comma up here. Did you notice these?
23:00:53
Beautiful. Okay. So now we can create our model by going torch or an instance of the tiny VGG to see if our model holds up. Let's create model zero equals tiny VGG. And I'm going to pass in the input shape. What is the input shape? It's going to be the number of color channels of our image. So number of color channels in our image data, which is three, because we have color images.
23:01:19
And then we're going to put in hidden units, equals 10, which will be the same number of hidden units as the tiny VGG architecture. One, two, three, four, five, six, seven, eight, nine, 10. Again, we could put in 10, we could put in 100, we could put in 64, which is a good multiple of eight. So let's just leave it at 10 for now. And then the output shape is going to be what? It's going to be the length of our class names, because we want one hidden unit or one output unit
23:01:49
per class. And then we're going to send it to the target device, which is of course CUDA. And then we can check out our model zero here. Beautiful. So that took a few seconds, as you saw there, to move to the GPU memory. So that's just something to keep in mind for when you build large neural networks and you want to speed up their computation, is to use operator fusion where you can, because as you saw, it took a few seconds for our model to just move from the CPU,
23:02:19
which is the default to the GPU. So we've got our architecture here. But of course, we know that this potentially is wrong. And how would we find that out? Well, we could find the right hidden unit shape or we could find that it's wrong by passing some dummy data through our model. So that's one of my favorite ways to troubleshoot a model. Let's in the next video pass some dummy data through our model and see if we've implemented the forward pass correctly. And also check the
23:02:49
input and output shapes of each of our layers. I'll see you there. In the last video, we replicated the tiny VGG architecture from the CNN explainer website, very similar to the model that we built in section 03. But this time, we're using color images instead of grayscale images. And we did it quite a bit faster than what we previously did, because we've already covered it, right? And you've had some experience now building pilotage models from scratch.
23:03:21
So we're going to pick up the pace when we build our models. But let's now go and try a dummy forward pass to check that our forward method is working correctly and that our input and output shapes are correct. So let's create a new heading. Try a forward pass on a single image. And this is one of my favorite ways to test the model. So let's first get a single image. Get a single image. We want an image batch. Maybe we get an image batch, get a single image batch, because
23:03:58
we've got images that are batches already image batch. And then we'll get a label batch. And we'll go next, it a train data loader. Simple. That's the data loader that we're working with for now.
23:04:16
And then we'll check image batch dot shape and label batch dot shape.
23:04:25
Wonderful. And now let's see what happens. Try a forward pass. Oh, I spelled single wrong up here. Try a forward pass. We could try this on a single image trying it on a same batch will result in similar results. So let's go model zero. And we're just going to pass it in the image batch and see what happens. Oh, no. Of course, we get that input type, torch float tensor and wait type torch CUDA float tensor should be the same or input should be.
23:05:00
So we've got tensors on a different device, right? So this is on the CPU, the image batch, whereas our model is, of course, on the target device. So we've seen this error a number of times. Let's see if this fixes it. Oh, we get an other error. And we kind of expected this type of error. We've got runtime error amount one and mat two shapes cannot be multiplied. 32. So that looks like the batch size 2560 and 10. Hmm, what is 10? Well, recall that 10 is the number of hidden
23:05:35
units that we have. So this is the size here. That's 10 there. So it's trying to multiply a matrix of this size by this size. So 10 has got something going on with it. We need to get these two numbers, the middle numbers, to satisfy the rules of matrix multiplication, because that's what happens in our linear layer. We need to get these two numbers the same.
23:06:00
And so our hint and my trick is to look at the previous layer. So if that's our batch size, where does this value come from? Well, could it be the fact that a tensor of this size goes through the flatten layer? Recall that we have this layer up here. So we've printed out the shape here of the conv block, the output of conv block one. Now this shape here is the output of conv block two. So we've got this number, the output of conv block one, and then the output of conv
23:06:36
block two. So that must be the input to our classifier layer. So if we go 10 times 16 times 16, what do we get? 2560. Beautiful. So we can multiply our hitting units 10 by 16 by 16, which is the shape here. And we get 2560. Let's see if that works. We'll go up here, times 16 times 16. And let's see what happens. We'll rerun the model, we'll rerun the image batch, and then we'll pass it. Oh, look at that. Our model works. Or the shapes at least line up. We don't know if it works
23:07:16
yet. We haven't started training yet. But this is the output size. We've got the output. It's on the CUDA device, of course. But we've got 32 samples with three numbers in each. Now these are going to be as good as random, because we haven't trained our model yet. We've only initialized it here with random weights. So we've got 32 or a batch worth of random predictions on 32 images. So you see how the output shape here three corresponds to the output shape we set up here.
23:07:47
Output shape equals length class names, which is exactly the number of classes that we're dealing with. But I think our number is a little bit different to what's in the CNN explainer 1616. How did they end up with 1313? You know what? I think we got one of these numbers wrong, kernel size, stride, padding. Let's have a look. Jump into here. If we wanted to truly replicate it,
23:08:14
is there any padding here? I actually don't think there's any padding here. So what if we go back here and see if we can change this to zero and change this to zero? Zero. I'm not sure if this will work, by the way. If it doesn't, it's not too bad, but we're just trying to line up the shapes with the CNN explainer to truly replicate it. So the output of the COM Block 1 should be 30-30-10. What are we working with at the moment? We've got 32-32-10. So let's see if removing the padding
23:08:46
from our convolutional layers lines our shape up with the CNN explainer. So I'm going to rerun this, rerun our model. I've set the padding to zero on all of our padding hyper parameters. Oh, and we get another error. We get another shape error. Of course we do, because we've now got different shapes. Wow, do you see how often that these errors come up?
23:09:10
Trust me, I spend a lot of time troubleshooting these shape errors. So we now have to line up these shapes. So we've got 13-13-10. Now does that equal 16-90? Let's try it out. 13-13-10.
23:09:24
16-90. Beautiful. And do our shapes line up with the CNN explainer? So we've got 30-30-10. Remember, these are in PyTorch. So color channels first, whereas this is color channels last. So yeah, we've got the output of our first COM Block is lining up here. That's correct. And then same with the second block. How good is that? We've officially replicated the CNN explainer model. So we can take this value 13-13-10 and bring it back up here. 13-13-10. Remember,
23:09:55
hidden units is 10. So we're just going to multiply it by 13-13. You could calculate these shapes by hand, but my trick is I like to let the error codes give me a hint of where to go. And boom, there we go. We get it working again. Some shape troubleshooting on the fly. So now we've done a single forward pass on the model. We can kind of verify that our data at least flows through it. What's next? Well, I'd like to show you another little package that I like to use
23:10:27
to also have a look at the input and output shapes of my model. And that is called Torch Info. So you might want to give this a shot before we go into the next video. But in the next video, we're going to see how we can use Torch Info to print out a summary of our model. So we're going to get something like this. So this is how beautifully easy Torch Info is to use. So give that a shot, install it into Google CoLab and run it in a cell here. See if you can get
23:10:57
something similar to this output for our model zero. And I'll see you in the next video. We'll try that together. In the last video, we checked our model by doing a forward pass on a single batch. And we learned that our forward method so far looks like it's intact and that we don't get any shape errors as our data moves through the model. But I'd like to introduce to you one of my favorite packages for finding out information from a PyTorch model. And that is Torch Info.
23:11:31
So let's use Torch Info to get an idea of the shapes going through our model. So you know how much I love doing things in a programmatic way? Well, that's what Torch Info does. Before, we used print statements to find out the different shapes going through our model. And I'm just going to comment these out in our forward method so that when we run this later on during training, we don't get excessive printouts of all the shapes. So let's see what Torch Info
23:12:00
does. And in the last video, I issued a challenge to give it a go. It's quite straightforward of how to use it. But let's see it together. This is the type of output we're looking for from our tiny VGG model. And of course, you could get this type of output from almost any PyTorch model. But we have to install it first. And as far as I know, Google CoLab doesn't come with Torch Info by default. Now, you might as well try this in the future and see if it works. But yeah, I don't
23:12:29
get this module because my Google CoLab instance doesn't have an install. No problem with that. Let's install Torch Info here. Install Torch Info and then we'll import it if it's available. So we're going to try and import Torch Info. If it's already installed, we'll import it. And then if it doesn't work, if that try block fails, we're going to run pip install Torch Info. And then we will import Torch Info. And then we're going to run down here from Torch Info,
23:13:06
import summary. And then if this all works, we're going to get a summary of our model. We're going to pass it in model zero. And we have to put in an input size here. Now that is an example of the size of data that will flow through our model. So in our case, let's put in an input size of 1, 3, 64, 64. So this is an example of putting in a batch of one image. You could potentially put in 32 here if you wanted, but let's just put in a batch of a singular image. And of course,
23:13:37
we could change these values here if we wanted to, 24 to 24. But what you might notice is that if it doesn't get the right input size, it produces an error. There we go. So just like we got before when we printed out our input sizes manually, we get an error here. Because what Torch Info behind the scenes is going to do is it's going to do a forward pass on whichever model you pass it with an input size of whichever input size you give it. So let's put in the input size that
23:14:06
our model was built for. Wonderful. So what Torch Info gives us is, oh, excuse me, we didn't comment out the printouts before. So just make sure we've commented out these printouts in the forward method of our 20 VGG class. So I'm just going to run this, then we run that, run that, just to make sure everything still works. We'll run Torch Info. There we go. So no printouts from our model, but this is, look how beautiful this is. I love how this prints out. So we have
23:14:40
our tiny VGG class, and then we can see it's comprised of three sequential blocks. And then inside those sequential blocks, we have different combinations of layers. We have some conv layers, some relu layers, some max pool layers. And then the final layer is our classification layer with a flatten and a linear layer. And we can see the shapes changing throughout our model. As our data goes in and gets manipulated by the various layers. So are these in line with
23:15:10
the CNN explainer? So if we check this last one, we've already verified this before.
23:15:17
And we also get some other helpful information down here, which is total params. So you can see that each of these layers has a different amount of parameters to learn. Now, recall that a parameter is a value such as a weight or a bias term within each of our layers, which starts off as a random number. And the whole goal of deep learning is to adjust those random numbers to better represent our data. So in our case, we have just over 8000 total parameters. Now this is actually quite small.
23:15:49
In the future, you'll probably play around with models that have a million parameters or more. And models now are starting to have many billions of parameters. And we also get some information here, such as how much the model size would be. Now this would be very helpful, depending on where we had to put our model. So what you'll notice is that as a model gets larger, as more layers, it will have more parameters, more weights and bias terms that can be adjusted
23:16:15
to learn patterns and data. But its input size and its estimated total size would definitely get bigger as well. So that's just something to keep in mind if you have size constraints in terms of storage in your future applications. So ours is under a megabyte, which is quite small. But you might find that some models in the future get up to 500 megabytes, maybe even over a gigabyte. So just keep that in mind for going forward. And that's the crux of torch info, one of my
23:16:44
favorite packages, just gives you an idea of the input and output shapes of each of your layers. So you can use torch info wherever you need. It should work with most of your PyTorch models. Just be sure to pass it in the right input size. You can also use it to verify like we did before, if the input and output shapes are correct. So check that out, big shout out to Tyler Yup, and everyone who's created the torch info package. Now in the next video, let's move towards training
23:17:13
our tiny VGG model. We're going to have to create some training and test functions. If you want to jump ahead, we've already done this. So I encourage you to go back to section 6.2 in the functionalizing training and test loops. And we're going to build functions very similar to this, but for our custom data set. So if you want to replicate these functions in this notebook, give that a go. Otherwise, I'll see you in the next video and we'll do it together.
23:17:43
How'd you go? Did you give it a shot? Did you try replicating the train step and the test step function? I hope you did. Otherwise, let's do that in this video, but this time we're going to do it for our custom data sets. And what you'll find is not much, if anything, changes, because we've created our train and test loop functions in such a way that they're generic. So we want to create a train step function. And by generic, I mean they can be used with almost any model and
23:18:16
data loader. So train step is takes in a model and data loader and trains the model on the data loader. And we also want to create another function called test step, which takes in a model and a data loader and other things and evaluates the model on the data loader. And of course, for the train step and for the test step, each of them respectively are going to take a training data loader. I just might make this a third heading so that our outline looks nice, beautiful.
23:18:54
Section seven is turning out to be quite a big section. Of course, we want them to be respectively taken their own data loader. So train takes in the train data loader, test takes in the test data loader. Without any further ado, let's create the train step function. Now we've seen this one in the computer vision section. So let's see what we can make here. So we need a train step, which is going to take in a model, which will be a torch and then dot module. And we want
23:19:21
it also to take in a data loader, which will be a torch dot utils dot data dot data loader.
23:19:29
And then it's going to take in a loss function, which is going to be a torch and then dot module as well. And then it's going to take in an optimizer, which is going to be torch opt in dot optimizer. Wonderful. And then what do we do? What's the first thing that we do in a training step? Well, we put the model in train mode. So let's go model dot train.
23:19:58
Then what shall we do next? Well, let's set up some evaluation metrics, one of them being loss and one of them being accuracy. So set up train loss and train accuracy values. And we're going to accumulate these per batch because we're working with batches. So we've got train loss and train act equals zero, zero. Now we can loop through our data loader. So let's write loop through
23:20:28
data loader. And we'll loop through each of the batches in this because we've batchified our data loader. So for batch x, y, in enumerate data loader, we want to send the data to the target device. So we could even put that device parameter up here. Device equals device. We'll set that to device by default. And then we can go x, y equals x dot two device. And y dot two device.
23:21:05
Beautiful. And now what do we do? Well, remember the pie torch, the unofficial pie torch optimization song, we do the forward pass. So y pred equals model om x. And then number two is we calculate the last. So calculate the loss. Let's go loss equals loss function. And we're going to pass it in y pred y. We've done this a few times now. So that's why we're doing it a little bit faster. So I hope you noticed that the things that we've covered before, I'm stepping up the pace a bit.
23:21:42
So it might be a bit of a challenge, but that's all right, you can handle it. And then, so that's accumulating the loss. So we're starting from zero up here. And then each batch, we're doing a forward pass, calculating the loss, and then adding it to the overall train loss. And so we're going to optimize a zero grad. So zero, the gradients of the optimizer for each new batch. And then we're going to perform back propagation. So loss backwards. And then five, what do we do? Optimize a step,
23:22:16
step, step. Wonderful. Look at that. Look at us coding a train loop in a minute or so. Now, let's calculate the accuracy and accumulate it. Calculate the, you notice that we don't have an accuracy function here. That's because accuracy is quite a straightforward metric to calculate. So we'll first get the, the y pred class, because this is going to output model logits. As we've seen before, the raw output of a model is logits. So to get the class, we're going to take
23:22:54
the arg max torch dot softmax. So we'll get the prediction probabilities of y pred, which is the raw logits, what we've got up here, across dimension one, and then also across dimension one here.
23:23:09
Beautiful. So that should give us the labels. And then we can find out if this is wrong by checking it later on. And then we're going to create the accuracy by taking the y pred class, checking for a quality with the right labels. So this is going to give us how many of these values equal true. And we want to take the sum of that, take the item of that, which is just a single integer. And then we want to divide it by the length of y pred. So we're just getting the
23:23:41
total number that are right, and dividing it by the length of samples. So that's the formula for accuracy. Now we can come down here outside of the batch loop, we know that because we've got this helpful line drawn here. And we can go adjust metrics to get the average loss and accuracy per batch. So we're going to set train loss is equal to train loss, divided by the length of the data loader. So the number of batches in total. And the train accuracy is the train
23:24:15
act, divided by the length of the data loader as well. So that's going to give us the average loss and average accuracy per epoch across all batches. So train act. Now that's a pretty good looking function to me for a train step. Do you want to take on the test step? So pause the video, give it a shot, and you'll get great inspiration from this notebook here. Otherwise, we're going to do it together in three, two, one, let's do the test step. So create a test step function.
23:24:53
So we want to be able to call these functions in an epoch loop. And that way, instead of writing out training and test code for multiple different models, we just write it out once, and we can call those functions. So let's create def test step, we're going to do model, which is going to be
23:25:11
if I could type torch and then module. And then we're going to do data loader, which is torch utils dot data, that data loader, capital L there. And then we're going to just pass in a loss function here, because we don't need an optimizer for the test function. We're not trying to optimize anything, we're just trying to evaluate how our model did on the training dataset. And let's put in the device here, why not? That way we can change the device if we need
23:25:42
to. So put model in a val mode, because we're going to be evaluating or we're going to be testing. Then we can set up test loss and test accuracy values. So test loss and test act. We're going to make these zero, we're going to accumulate them per batch. But before we go through the batch, let's turn on inference mode. So this is behind the scenes going to take care of a lot of pie torch functionality that we don't need. That's very helpful during training, such as tracking gradients.
23:26:16
But during testing, we don't need that. So loop through data loader or data batches. And we're going to go for batch x, y in enumerate data loader. You'll notice that above, we didn't actually use this batch term here. And we probably won't use it here either. But I just like to go through and have that there in case we wanted to use it anyway. So send data to the target device. So we're going to go x, y equals x dot two device. And same with y dot two device. Beautiful. And
23:26:59
then what do we do for an evaluation step or a test step? Well, of course, we do the forward pass, forward pass. And we're going to, let's call these test pred logits and get the raw outputs of our model. And then we can calculate the loss on those raw outputs, calculate the loss. We get the loss is equal to loss function on test pred logits versus y. And then we're going to accumulate the loss. So test loss plus equals loss dot item. Remember, item just gets a single integer from
23:27:39
whatever term you call it on. And then we're going to calculate the accuracy. Now we can do this exactly how we've done for the training data set or the training step. So test pred labels, we're going to, you don't, I just want to highlight the fact that you actually don't need to take the softmax here, you could just take the argmax directly from this. The reason why we take the softmax. So you could do the same here, you could just directly take the argmax of the logits. The
23:28:11
reason why we get the softmax is just for completeness. So if you wanted the prediction probabilities, you could use torch dot softmax on the prediction logits. But it's not 100% necessary to get the same values. And you can test this out yourself. So try this with and without the softmax and see if you get the same results. So we're going to go test accuracy. Plus equals, now we'll just create our accuracy calculation on the fly test pred labels. We'll check for equality on the y,
23:28:41
then we'll get the sum of that, we'll get the item of that, and then we'll divide that by the length of the test pred labels. Beautiful. So it's going to give us accuracy per batch. And so now we want to adjust the metrics to get average loss and accuracy per batch. So test loss equals test loss divided by length of the data loader. And then we're going to go test, ac equals test, act divided by length of the data loader. And then finally, we're going to
23:29:17
return the test loss, not lost, and test accuracy. Look at us go. Now, in previous videos, that took us, or in previous sections, that took us a fairly long time. But now we've done it in about 10 minutes or so. So give yourself a pat in the back for all the progress you've been making. But now let's in the next video, we did this in the computer vision section as well. We created, do we create a train function? Oh, no, we didn't. But we could. So let's create a function to
23:29:53
functionize this. We want to train our model. I think we did actually. Deaf train, we've done so much. I'm not sure what we've done. Oh, okay. So looks like we might not have. But in the next video, give yourself this challenge, create a function called train that combines these two functions and loops through them both with an epoch range. So just like we've done here in the previous notebook, can you functionize this? So just this step here. So you'll need to take in a
23:30:26
number of epochs, you'll need to take in a train data loader and a test data loader, a model, a loss function, an optimizer, and maybe a device. And I think you should be pretty on your way to all the steps we need for train. So give that a shot. But in the next video, we're going to create a function that combines train step and test step to train a model. I'll see you there.
23:30:51
How'd you go? In the last video, I issued you the challenge to combine our train step function, as well as our test step function together in their own function so that we could just call one function that calls both of these and train a model and evaluate it, of course. So let's now do that together. I hope you gave it a shot. That's what it's all about. So we're going to create a train function. Now the role of this function is going to, as I said, combine
23:31:18
train step and test step. Now we're doing all of this on purpose, right, because we want to not have to rewrite all of our code all the time. So we want to be functionalizing as many things as possible, so that we can just import these later on, if we wanted to train more models and just leverage the code that we've written before, as long as it works. So let's see if it does, we're going to create a train function. I'm going to first import TQDM, TQDM.auto,
23:31:48
because I'd like to get a progress bar while our model is training. There's nothing quite like watching a neural network train. So step number one is we need to create a train function that takes in various model parameters, plus optimizer, plus data loaders, plus a loss function. A whole bunch of different things. So let's create def train. And I'm going to pass in a model here, which is going to be torch and then dot module. You'll notice that the inputs of this are going
23:32:25
to be quite similar to our train step and test step. I don't actually need that there.
23:32:32
So we also want a train data loader for the training data, torch dot utils dot data dot data loader. And we also want a test data loader, which is going to be torch dot utils dot data dot data loader. And then we want an optimizer. So the optimizer will only be used with our training data set, but that's okay. We can take it as an input of the miser. And then we want a loss function. This will generally be used for both our training and testing step. Because that's
23:33:04
what we're combining here. Now, since we're working with multi class classification, I'm going to set our loss function to be a default of an n dot cross entropy loss.
23:33:15
Then I'm going to get epochs. I'm going to set five, we'll train for five epochs by default. And then finally, I'm going to set the device equal to the device. So what do we get wrong here?
23:33:30
That's all right. We'll just keep coding. We'll ignore these little red lines. If they stay around, we'll come back to them. So step number two, I'm going to create. This is a step you might not have seen, but I'm going to create an empty results dictionary. Now, this is going to help us track our results. Do you recall in a previous notebook, we outputted a model dictionary for how a model went. So if we look at model one results, yeah, we got a dictionary like this.
23:33:57
So I'd like to create one of these on the fly, but keep track of the result every epoch. So what was the loss on epoch number zero? What was the accuracy on epoch number three? So we'll show you how I'll do that. We can use a dictionary and just update that while our model trains. So results, I want to keep track of the train loss. So we're going to set that equal to an empty list and just append to it. I also want to keep track of the train accuracy. We'll set that as
23:34:25
an empty list as well. I also want to keep track of the test loss. And I also want to keep track of the test accuracy. Now, you'll notice over time that these, what you can track is actually very flexible. And what your functions can do is also very flexible. So this is not the gold standard of doing anything by any means. It's just one way that works. And you'll probably find in the future that you need different functionality. And of course, you can code that out. So let's
23:34:57
now loop through our epochs. So for epoch in TQDM, let's create a range of our epochs above. And then we can set the train loss. Have I missed a comma up here somewhere? Type annotation not supported for that type of expression. Okay, that's all right. We'll just leave that there. So we're going to go train loss and train act, recall that our train step function that we created in the previous video, train step returns our train loss and train act. So as I
23:35:30
said, I want to keep track of these throughout our training. So I'm going to get them from train step. Then for each epoch in our range of epochs, we're going to pass in our model and perform a training step. So the data loader here is of course going to be the train data loader. The loss function is just going to be the loss function that we pass into the train function. And then the optimizer is going to be the optimizer. And then the device is going to be device.
23:36:00
Beautiful. Look at that. We just performed a training step in five lines of code. So let's keep pushing forward. It's telling us we've got a whole bunch of different things here. Epox is not defined. Maybe we just have to get rid of this. We can't have the type annotation here. And that'll that'll stop. That'll stop Google Colab getting angry at us. If it does anymore, I'm just going to ignore it for now. Epox. Anyway, we'll leave it at that. We'll find out if there's
23:36:28
an error later on. Test loss. You might be able to find it before I do. So test step. We're going to pass in the model. We're going to pass in a data loader. Now this is going to be the test data loader. Look at us go. Grading training and test step functions, loss function. And then we don't need an optimizer. We're just going to pass in the device. And then behind the scenes, both of these functions are going to train and test our model. How cool is that? So still within
23:36:56
the loop. This is important. Within the loop, we're going to have number four is we're going to print out. Let's print out what's happening. Print out what's happening. We can go print. And we'll do a fancy little print statement here. We'll get the epoch. And then we will get the train loss, which will be equal to the train loss. We'll get that to, let's go four decimal places. How about that? And then we'll get the train accuracy, which is going to be the
23:37:26
train act. We'll get that to four, maybe three decimal of four, just for just so it looks nice. It looks aesthetic. And then we'll go test loss. We'll get that coming out here. And we'll pass in the test loss. We'll get that to four decimal places as well. And then finally, we'll get the test accuracy. So a fairly long print statement here. But that's all right. We'd like to see how our model is doing while it's training. Beautiful. And so again, still within the epoch, we want to
23:37:59
update our results dictionary so that we can keep track of how our model performed over time. So let's pass in results. We want to update the train loss. And so this is going to be this. And then we can append our train loss value. So this is just going to expend the list in here with the train loss value, every epoch. And then we'll do the same thing on the train accuracy, append train act. And then we'll do the same thing again with test loss dot append test loss.
23:38:41
And then we will finally do the same thing with the test accuracy test accuracy. Now, this is a pretty big function. But this is why we write the code now so that we can use it multiple times later on. So return the field results at the end of the epoch. So outside the epochs loop. So our loop, we're outside it now. Let's return results. Now, I've probably got an error somewhere here and you might be able to spot it. Okay, train data loader. Where do we get
23:39:16
that invalid syntax? Maybe up here, we don't have a comma here. Was that the issue the whole time? Wonderful. You might have seen that I'm completely missed that. But we now have a train function to train our model. And the train function, of course, is going to call out our train step function and our test step function. So what's left to do? Well, nothing less than train and evaluate model zero. So our model is way back up here. How about in the next video, we leverage
23:39:50
our functions, namely just the train function, because it's going to call our train step function and our test step function and train our model. So I'm going to encourage you to give that a go. You're going to have to go back to the workflow. Maybe you'll maybe already know this.
23:40:06
So what have we done? We've got our data ready and we turned it into tenses using a combination of these functions. We've built and picked a model while we've built a model, which is the tiny VGG architecture. Have we created a loss function yet? I don't think we have or an optimizer. I don't think we've done that yet. We've definitely built a training loop though.
23:40:28
We aren't using torch metrics. We're just using accuracy, but we could use this if we want. We haven't improved through experimentation yet, but we're going to try this later on and then save and reload the model. We've seen this before. So I think we're up to picking a loss function and an optimizer. So give that a shot. In the next video, we're going to create a loss function and an optimizer and then leverage the functions we've spent in the last two videos
23:40:54
creating to train our first model model zero on our own custom data set. This is super exciting. I'll see you in the next video.
23:41:06
Who's ready to train and evaluate model zero? Put your hand up. I definitely am. So let's do it together. We're going to start off section 7.7 and we're going to put in train and evaluate model zero, our baseline model on our custom data set. Now, if we refer back to the PyTorch workflow, I issued you the challenge in the last video to try and create a loss function and an optimizer. I hope you gave that a go, but we've already built a
23:41:34
training loop. So we're going to leverage our training loop functions, namely train, train step and test step. All we need to do now is instantiate a model, choose a loss function and an optimizer and pass those values to our training function. So let's do that. All right, this is so exciting. Let's set the random seeds. I'm going to set torch manual seed 42 and torch cuda manual seed 42. Now remember, I just want to highlight something. I read an article the other day about not using
23:42:09
random seeds. The reason why we are using random seeds is for educational purposes. So to try and get our numbers on my screen and your screen as close as possible, but in practice, you quite often don't use random seeds all the time. The reason why is because you want your models performance to be similar regardless of the random seed that you use. So just keep that in mind going forward. We're using random seeds to just exemplify how we can get similar numbers on our page. But
23:42:41
ideally, no matter what the random seed was, our models would go in the same direction. That's where we want our models to eventually go. But we're going to train for five epochs. And now let's create a recreate an instance of tiny VGG. We can do so because we've created the tiny VGG class. So tiny VGG, which is our model zero. We don't have to do this, but we're going to do it any later. So we've got all the code in one place, tiny VGG. What is our input shape
23:43:13
going to be? That is the number of color channels of our target images. And because we're dealing with color images, we have an input shape of three. Previously, we used an input shape of one to deal with grayscale images. I'm going to set hidden units to 10 in line with the CNN explainer website. And the output shape is going to be the number of classes in our training data set. And then, of course, we're going to send the target model to the target device. So what do we do now?
23:43:47
Well, we set up a loss function and an optimizer, loss function, and optimizer. So our loss function is going to be because we're dealing with multiclass classification, and then cross entropy, if I could spell cross entropy loss. And then we're going to have an optimizer. This time, how about we mix things up? How about we try the atom optimizer? Now, of course, the optimizer is one of the hyper parameters that you can set for your model,
23:44:14
and a hyper parameter being a value that you can set yourself. So the parameters that we want to optimize are our model zero parameters. And we're going to set a learning rate of 0.001. Now, recall that you can tweet this learning rate, if you like, but I believe, did I just see that the default learning rate of atom is 0.001? Yeah, there we go. So Adam's default learning rate is one to the power of 10 to the negative three. And so that is a default learning rate for Adam.
23:44:49
And as I said, oftentimes, different variables in the pytorch library, such as optimizers, have good default values that work across a wide range of problems. So we're just going to stick with the default. If you want to, you can experiment with different values of this.
23:45:07
But now let's start the timer, because we want to time our models.
23:45:13
We're going to import from time it. We want to get the default timer class. And I'm going to import that as timer, just so we don't have to type out default timer. So the start time is going to be timer. This is going to just put a line in the sand of what the start time is at this particular line of code. It's going to measure that. And then we're going to train model zero. Now this is using, of course, our train function. So let's write model zero results, and then
23:45:44
they wrote model one, but we're not up to there yet. So let's go train model equals model zero. And this is just the training function that we wrote in a previous video. And the train data is going to be our train data loader. And we've got train data loader simple, because we're not using data augmentation for model one. And then our test data loader is going to be our test data loader simple. And then we're going to set our optimizer, which is equal to the optimizer we just
23:46:15
created. Friendly atom optimizer. And the loss function is going to be the loss function that we just created, which is an n cross entropy loss. Finally, we can send in epochs is going to be num epochs, which is what we set at the start of this video to five. And of course, we could train our model for longer if we wanted to. But the whole idea of when you first start training a model is to keep your experiments quick. So that's why we're only training for five, maybe later on you
23:46:44
train for 10, 20, tweak the learning rate, do a whole bunch of different things. But let's go down here, let's end the timer, see how long our models took to train, and the timer and print out
23:46:58
how long it took. So in a previous section, we created a helper function for this. We're just going to simplify it in this section. And we're just going to print out how long the training time was. Total training time. Let's go n time minus start time. And then we're going to go point, we'll take it to three decimal places, hey, seconds, you ready to train our first model, our first convolutional neural network on our own custom data set on pizza, stake and sushi
23:47:33
images. Let's do it. You're ready? Three, two, one, no errors. Oh, there we go. Okay, should this be trained data loader? Did you notice that? What is our trained data taker's input? Oh, we're not getting a doc string. Oh, there we go. We want trained data loader, data loader, and same with this, I believe. Let's try again. Beautiful. Oh, look at that lovely progress bar. Okay, how's our model is training quite fast? Okay. All right, what do we
23:48:10
get? So we get an accuracy on the training data set of about 40%. And we get an accuracy on the test data set of about 50%. Now, what's that telling us? It's telling us that about 50% of the time our model is getting the prediction correct. But we've only got three classes. So even if our model was guessing, it would get things right 33% of the time. So even if you just guessed pizza every single time, because we only have three classes, if you guessed pizza every single time, you get
23:48:46
a baseline accuracy of 33%. So our model isn't doing too much better than our baseline accuracy. Of course, we'd like this number to go higher, and maybe it would if it trained for longer. So I'll let you experiment with that. But if you'd like to see some different methods of improving a model, recall back in section number O two, we had an improving a model section, improving a model. Here we go. So here's some things you might want to try.
23:49:18
We can improve a model by adding more layers. So if we come back to our tiny VGG architecture,
23:49:25
right up here, we're only using two convolutional blocks. Perhaps you wanted to add in a convolutional block three. You can also add more hidden units. Right now we're using 10 hidden units. You might want to double that and see what happens. Fitting for longer. This is what we just spoke about. So right now we're only fitting for five epochs. So if you maybe wanted to try double that again, and then even double that again, changing the activation functions. So maybe relu is not the
23:49:57
ideal activation function for our specific use case. Change the learning rate. We've spoken about that before. So right now our learning rate is 0.001 for Adam, which is the default. But perhaps there's a better learning rate out there. Change the loss function. This is probably not
23:50:15
in our case, not going to help too much because cross entropy loss is a pretty good loss for multi class classification. But these are some things that you could try these first three, especially. You could try quite quickly. You could try doubling the layers. You could try adding more hidden units. And you could try fitting for longer. So I'd give that a shot. But in the next video, we're going to take our model zero results, which is a dictionary or at
23:50:42
least it should be. And we're going to plot some loss curves. So this is a good way to inspect how our model is training. Yes, we've got some values here. Let's plot these in the next video. I'll see you there. In the last video, we trained our first convolutional neural network on custom data. So you should be very proud of that. That is no small feat to take our own data set of whatever we want and train apply to its model on it. However, we did find that it didn't perform as well as we'd
23:51:17
like it to. We also highlighted a few different things that we could try to do to improve it. But now let's plot our models results using a loss curve. So I'm going to write another heading down here. We'll go, I believe we're up to 7.8. So plot the loss curves of model zero. So what is a loss curve? So I'm going to write down here, a loss curve is a way of tracking your models progress over time. So if we just looked up Google and we looked up loss curves,
23:51:56
oh, there's a great guide by the way. I'm going to link this. But I'd rather if and doubt code it out than just look at guides. Yeah, loss curves. So yeah, loss over time. So there's our loss value on the left. And there's say steps, which is epochs or batches or something like that. Then we've got a whole bunch of different loss curves over here. Essentially, what we want it to do is go down over time. So that's the idea loss curve. Let's go back down here.
23:52:28
And a good guide for different loss curves can be seen here. We're not going to go through that just yet. Let's focus on plotting our own models, loss curves, and we can inspect those. Let's get the model keys. Get the model zero results keys. I'm going to type in model zero results dot keys because it's a dictionary. Let's see if we can write some code to plot these
23:52:59
values here. So yeah, over time. So we have one value for train loss, train, act, test loss, and test act for every epoch. And of course, these lists would be longer if we train for more epochs. But let's just how about we create a function called def plot loss curves, which will take in a results dictionary, which is of string and a list of floats. So this just means that our results parameter here is taking in a dictionary that has a string as a key.
23:53:33
And it contains a list of floats. That's what this means here. So let's write a doc string plots training curves of a results dictionary. Beautiful. And so we're in this section of our workflow, which is kind of like a, we're kind of doing something similar to TensorBoard, what it does. I'll let you look into that if you want to. Otherwise, we're going to see it later on. But we're really evaluating our model here. Let's write some plotting code. We're going to use map plot
23:54:08
lib. So we want to get the lost values of the results dictionary. So this is training and test.
23:54:19
Let's set loss equal to results train loss. So this is going to be the loss on the training data set. And then we'll create the test loss, which is going to be, well, index on the results dictionary and get the test loss. Beautiful. Now we'll do the same and we'll get the accuracy. Get the accuracy values of the results dictionary. So training and test.
23:54:44
Then we're going to go accuracy equals results. This will be the training accuracy train act and accuracy. Oh, we'll call this test accuracy actually test accuracy equals results test act. Now let's create a number of epochs. So we want to figure out how many epochs we did. We can do that by just counting the length of this value here. So figure out how many epochs there were. So we'll set epochs equal to a range because we want to plot it over time. Our models results
23:55:18
over time. That's that's the whole idea of a loss curve. So we'll just get the the length of our results here. And we'll get the range. So now we can set up a plot.
23:55:34
Let's go PLT dot figure. And we'll set the fig size equal to something nice and big because we're going to do four plots. We want one for maybe two plots, one for the loss, one for the accuracy. And then we'll go plot the loss. PLT dot subplot. We're going to create one row, two columns, and index number one. We want to put PLT dot plot. And here's where we're going to plot the training loss. So we get that a label of train loss. And then we'll add another plot with epochs
23:56:12
and test loss. The label here is going to be test loss. And then we'll add a title, which will be loss PLT. Let's put a label on the X, which will be epochs. So we know how many steps we've done. This plot over here, loss curves, it uses steps. I'm going to use epochs. They mean almost the same thing. It depends on what scale you'd like to see your loss curves. We'll get a legend as well so that we are the labels appear. Now we're going to plot the accuracy. So PLT dot subplot.
23:56:49
Let's go one, two, and then index number two that this plot's going to be on PLT dot plot. We're going to go epochs accuracy. And the label here is going to be train accuracy.
23:57:04
And then we'll get on the next plot, which is actually going to be on the same plot. We'll put the test accuracy. That way we have the test accuracy and the training accuracy side by side, test accuracy same with the train loss and train, sorry, test loss. And then we'll give our plot a title. This plot is going to be accuracy. And then we're going to give it an X label, which is going to be epochs as well. And then finally, we'll get the plot, but legend,
23:57:31
a lot of plotting code here. But let's see what this looks like. Hey, if we've done it all right, we should be able to pass it in a dictionary just like this and see some nice plots like this.
23:57:44
Let's give it a go. And I'm going to call plot loss curves. And I'm going to pass in model 0 results.
23:57:58
All righty then. Okay. So that's not too bad. Now, why do I say that? Well, because we're looking here for mainly trends, we haven't trained our model for too long. Quantitatively, we know that our model hasn't performed at the way we'd like it to do. So we'd like the accuracy on both the train and test data sets to be higher. And then of course, if the accuracy is going higher, then the loss is going to come down. So the ideal trend for a loss curve is to go down from
23:58:31
the top left to the bottom right. In other words, the loss is going down over time. So that's, the trend is all right here. So potentially, if we train for more epochs, which I'd encourage you to give it a go, our model's loss might get lower. And the accuracy is also trending in the right way. Our accuracy, we want it to go up over time. So if we train for more epochs, these curves may continue to go on. Now, they may not, they, you never really know, right? You can guess these
23:59:02
things. But until you try it, you don't really know. So in the next video, we're going to have a look at some different forms of loss curves. But before we do that, I'd encourage you to go through this guide here, interpreting loss curves. So I feel like if you just search out loss curves, you're going to find Google's guide, or you could just search interpreting loss curves. Because as you'll see, there's many different ways that loss curves can be interpreted. But the ideal
23:59:29
trend is for the loss to go down over time, and metrics like accuracy to go up over time. So in the next video, let's cover a few different forms of loss curves, such as the ideal loss curve, what it looks like when your model's underfitting, and what it looks like when your model's overfitting. And if you'd like to have a primer on those things, I'd read through this guide here. Don't worry too much if you're not sure what's happening. We're going to cover a bit
23:59:57
more about loss curves in the next video. I'll see you there. In the last video, we looked at our model's loss curves, and also the accuracy curves. And a loss curve is a way to evaluate a model's performance over time, such as how long it was training for. And as you'll see, if you Google some images of loss curves, you'll see many different types of loss curves. They come in all different shapes and sizes. And there's many different ways to interpret loss curves. So
24:00:29
this is Google's testing and debugging and machine learning guide. So I'm going to set this as actually curriculum for this section. So we're up to number eight. Let's have a look at what should an ideal loss curve look like. So we'll just link that in there. Now, loss curve, I'll just rewrite here, is a loss curve is, I'll just make some space. A loss curve is one of the most helpful ways to troubleshoot a model. So the trend of a loss curve, you want it to go down over time,
24:01:15
and the trend typically of an evaluation metric, like accuracy, you want it to go up over time. So let's go into the keynote, loss curves. So a way to evaluate your model's performance over time. These are three of the main different forms of loss curve that you'll face. But again, there's many different types as mentioned in here, interpreting loss curves. Sometimes you get it going all over the place. Sometimes your loss will explode. Sometimes your metrics will be
24:01:45
contradictory. Sometimes your testing loss will be higher than your training loss. We'll have a look at what that is. Sometimes your model gets stuck. In other words, the loss doesn't reduce. Let's have a look at some loss curves here in the case of underfitting, overfitting, and just right. So this is the Goldilocks zone. Underfitting is when your model's loss on the training and test data sets could be lower. So in our case, if we go back to our loss curves, of course,
24:02:13
we want this to be lower, and we want our accuracy to be higher. So from our perspective, it looks like our model is underfitting. And we would probably want to train it for longer, say, 10, 20 epochs to see if this train continues. If it keeps going down, it may stop underfitting. So underfitting is when your loss could be lower. Now, the inverse of underfitting is called overfitting. And so two of the biggest problems in machine learning is trying to
24:02:45
underfitting. So in other words, make your loss lower and also reduce overfitting. These are both active areas of research because you always want your model to perform better, but you also want it to perform pretty much the same on the training set as it does the test set. And so overfitting would be when your training loss is lower than your testing loss. And why would this be overfitting? So it means overfitting because your model is essentially learning the
24:03:14
training data too well. And that means the loss goes down on the training data set, which is typically a good thing. However, this learning is not reflected in the testing data set. So your model is essentially memorizing patterns in the training data set that don't generalize well to the test data set. So this is where we come to the just right curve is that we want, ideally, our training loss to reduce as much as our test loss. And quite often, you'll find
24:03:45
that the loss is slightly lower on the training set than it is on the test set. And that's just because the model is exposed to the training data, and it's never seen the test data before. So it might be a little bit lower on the training data set than on the test data set. So underfitting, the model's loss could be lower. Overfitting, the model is learning the training data too well. Now, this would be equivalent to say you were studying for a final exam,
24:04:13
and you just memorize the course materials, the training set. And when it came time to the final exam, because you don't even memorize the course materials, you couldn't adapt those skills to questions you hadn't seen before. So the final exam would be the test set. So that's overfitting. The train loss is lower than the test loss. And just right, ideally, you probably won't see loss curves this exact smooth. I mean, they might be a little bit jumpy. Ideally, your training loss
24:04:44
and test loss go down at a similar rate. And of course, there's more combinations of these. If you'd like to see them, check out the Google's loss curve guide that you can check that out there. That's some extra curriculum. Now, you probably want to know how do you deal with underfitting and overfitting? Let's look at a few ways. We'll start with overfitting. So we want to reduce overfitting. In other words, we want our model to perform just as
24:05:10
well on the training data set as it does on the test data set. So one of the best ways to reduce overfitting is to get more data. So this means that our training data set will be larger. Our model will be exposed to more examples. And with us, in theory, it doesn't always work. These all come with a caveat, right? They don't always work as with many things in machine learning. So get more data, give your model more chance to learn patterns, generalizable patterns in a
24:05:38
data set. You can use data augmentation. So make your models training data set harder to learn. So we've seen a few examples of data augmentation. You can get better data. So not only more data, perhaps the data that you're using isn't that the quality isn't that good. So if you enhance the quality of your data set, your model may be able to learn better, more generalizable patterns and in turn reduce overfitting. Use transfer learning. So we're going to cover this in a later section
24:06:08
of the course. But transfer learning is taking one model that works, taking its patterns that it's learned and applying it to your own data set. So for example, I'll just go into the Torch Vision models library. Many of these models in here in Torch Vision, the models module, have already been trained on a certain data set and such as ImageNet. And you can take the weights or the patterns that these models have learned. And if they work well on an ImageNet data set, which is
24:06:41
millions of different images, you can adjust those patterns to your own problem. And oftentimes that will help with overfitting. If you're still overfitting, you can try to simplify your model. Usually this means taking away things like extra layers, taking away more hidden units. So say you had 10 layers, you might reduce it to five layers. Why does this? What's the theory behind this? Well, if you simplify your model and take away complexity from your model, you're kind of telling
24:07:10
your model, hey, use what you've got. And you're going to have to, because you've only got five layers now, you're going to have to make sure that those five layers work really well, because you've no longer got 10. And the same for hidden units. Say you started with 100 hidden units per layer, you might reduce that to 50 and say, hey, you had 100 before. Now use those 50 and make your patterns generalizable. Use learning rate decay. So the learning rate is how much your optimizer
24:07:41
updates your model's weight every step. So learning rate decay is to decay the learning rate over time. So you might look this up, you can look this up, go high torch, learning rate, scheduling. So what this means is you want to decrease your learning rate over time. Now, I know I'm giving you a lot of different things here, but you've got this keynote as a reference. So you can come across these over time. So learning rate scheduling. So we might look
24:08:15
into here, do we have schedule, scheduler, beautiful. So this is going to adjust the learning rate over time. So for example, at the start of when a model is training, you might want a higher learning rate. And then as the model starts to learn patterns more and more and more, you might want to reduce that learning rate over time so that your model doesn't update its patterns too much in later epochs. So that's the concept of learning rate scheduling. At the closer you get to convergence,
24:08:46
the lower you might want to set your learning rate, think of it like this. If you're reaching for a coin at the back of a couch, can we get an image of that coin at back of couch? Images. So if you're trying to reach a coin in the cushions here, so the closer you get to that coin, at the beginning, you might take big steps. But then the closer you get to that coin, the smaller the step you might take to pick that coin out. Because if you take a big step when you're really
24:09:15
close to the coin here, the coin might fall down the couch. The same thing with learning rate decay. At the start of your model training, you might take bigger steps as your model works its way down the loss curve. But then you get closer and closer to the ideal position on the loss curve. You might start to lower and lower that learning rate until you get right very close to the end and you can pick up the coin. Or in other words, your model can converge. And then finally, use
24:09:43
early stopping. So if we go into an image, is there early stopping here? Early stopping. Loss curves early stopping. So what this means is that you stop. Yeah, there we go. So there's heaps of different guides early stopping with PyTorch. Beautiful. So what this means is before your testing error starts to go up, you keep track of your model's testing error. And then you stop your model from training or you save the weight or you save the patterns where your model's loss
24:10:16
was the lowest. So then you could just set your model to train for an infinite amount of training steps. And as soon as the testing error starts to increase for say 10 steps in a row, you go back to this point here and go, I think that was where our model was the best. And the testing error started to increase after that. So we're going to save that model there instead of the model here. So that's the concept of early stopping. So that's dealing with overfitting. There are
24:10:42
other methods to deal with underfitting. So recall underfitting is when we have a loss that isn't as low as we'd like it to be. Our model is not fitting the data very well. So it's underfitting. So to reduce underfitting, you can add more layers slash units to your model. You're trying to increase your model's ability to learn by adding more layers or units. You can again tweak the learning rate. Perhaps your learning rate is too high to begin with and your model doesn't learn
24:11:09
very well. So you can adjust the learning rate again, just like we discussed with reaching for that coin at the back of a couch. If your model is still underfitting, you can train for longer. So that means giving your model more opportunities to look at the data. So more epochs, that just means it's got looking at the training set over and over and over and over again and trying to learn those patterns. However, you might find again, if you try to train for too long, your testing
24:11:35
error will start to go up. Your model might start overfitting if you train too long. So machine learning is all about a balance between underfitting and overfitting. You want your model to fit quite well. And so this is a great one. So you want your model to start fitting quite well. But then if you try to reduce underfitting too much, you might start to overfit and then vice versa, right? If you try to reduce overfitting too much, your model might underfit. So this is one of the
24:12:03
most fun dances in machine learning, the balance between overfitting and underfitting. Finally, you might use transfer learning. So transfer learning helps with overfitting and underfitting. Recall transfer learning is using a model's learned patterns from Ron problem and adjusting them to your own. We're going to see this later on in the course. And then finally, use less regularization. So regularization is holding your model back. So it's trying
24:12:29
to prevent overfitting. So if you do too much preventing of overfitting, in other words, regularizing your model, you might end up underfitting. So if we go back, we have a look at the ideal curves, underfitting. If you try to prevent underfitting too much, so increasing your model's capability to learn, you might end up overfitting. And if you try to prevent overfitting too much, you might end up underfitting. We are going for the just right section. And this is going to be a
24:12:57
balance between these two throughout your entire machine learning career. In fact, it's probably the most prevalent area of research is trying to get models not to underfit, but also not to overfit. So keep that in mind. A loss curve is a great way to evaluate your model's performance over time. And a lot of what we do with the loss curves is try to work out whether our model is underfitting or overfitting, and we're trying to get to this just right curve. We might not get
24:13:28
exactly there, but we want to keep trying getting as close as we can. So with that being said, let's now build another model in the next video. And we're going to try a method to try and see if we can use data augmentation to prevent our model from overfitting. Although that experiment doesn't sound like the most ideal one we could do right now, because it looks like our model is underfitting. So with your knowledge of what you've just learned in the previous video,
24:13:54
how to prevent underfitting, what would you do to increase this model's capability of learning patterns in the training data set? Would you train it for longer? Would you add more layers? Would you add more hidden units? Have a think and we'll start building another model in the next video. Welcome back. In the last video, we covered the important concept of a loss curve and how it can give us information about whether our model is underfitting. In other words, our model's loss
24:14:28
could be lower or whether it's overfitting. In other words, the training loss is lower than the test loss or far lower than the validation loss. That's another thing to note here is that I put training and test sets here. You could also do this with a validation data set and that the just right, the Goldilocks zone, is when our training and test loss are quite similar over time.
24:14:53
Now, there was a fair bit of information in that last video, so I just wanted to highlight that you can get this all in section 04, which is the notebook that we're working on. And then if you come down over here, if we come to section 8, watch an ideal loss curve look like we've got underfitting, overfitting, just right, how to deal with overfitting. We've got a few options here. We've got how to deal with underfitting and then we've got a few options there. And then if we
24:15:18
wanted to look for more, how to deal with overfitting. You could find a bunch of resources here and then how to deal with underfitting. You could find a bunch of resources here as well. So that is a very fine line, very fine balance that you're going to experience throughout all of your machine learning career. But it's time now to move on. We're going to move on to creating another model, which is tiny VGG, with data augmentation this time. So if we go back to the slide,
24:15:54
data augmentation is one way of dealing with overfitting. Now, it's probably not the most ideal experiment that we could take because our model zero, our baseline model, looks like it's underfitting. But data augmentation, as we've seen before, is a way of manipulating images to artificially increase the diversity of your training data set without collecting more data. So we could take our photos of pizza, sushi, and steak and randomly rotate them 30 degrees
24:16:24
and increase diversity forces a model to learn or hopefully learn. Again, all of these come with a caveat of not always being the silver bullet to learn more generalizable patterns. Now, I should have spelled generalizable here rather than generalization, but similar thing.
24:16:42
Let's go here. Let's create to start off with, we'll just write down.
24:16:48
Now let's try another modeling experiment. So this is in line with our PyTorch workflow, trying a model and trying another one and trying another one, so and so over again. This time, using the same model as before, but with some slight data augmentation. Oh, maybe we're not slight. That's probably not the best word. We'll just say with some data augmentation. And if we come down here, we're going to write section 9.1. We need to first
24:17:20
create a transform with data augmentation. So we've seen what this looks like before. We're going to use the trivial augment data augmentation, create training transform, which is, as we saw in a previous video, what PyTorch the PyTorch team have recently used to train their state-of-the-art computer vision models. So train transform trivial. This is what I'm going to call my transform. And I'm just going to from Torch Vision import
24:17:53
transforms. We've done this before. We don't have to re-import it, but I'm going to do it anyway, just to show you that we're re-importing or we're using transforms. And we're going to compose a transform here. Recall that transforms help us manipulate our data. So we're going to transform our images into size 64 by 64. Then we're going to set up a trivial augment transforms, just like we did before, trivial augment wide. And we're going to set the number of magnitude
24:18:24
bins here to be 31, which is the default here, which means we'll randomly use some data augmentation on each one of our images. And it will be applied at a magnitude of 0 to 31, also randomly selected. So if we lower this to five, the upper bound of intensity of how much that data augmentation is applied to a certain image will be less than if we set it to say 31. Now, our final transform here is going to be too tensor because we want our images in tensor format for our model.
24:18:57
And then I'm going to create a test transform. I'm going to call this simple, which is just going to be transforms dot compose. And all that it's going to have, oh, I should put a list here, all that it's going to have, we'll just make some space over here, is going to be transforms. All we want to do is resize the image size equals 64 64. Now we don't apply data augmentation to the test data set, because we only just want to evaluate our models on the test data set.
24:19:33
Our models aren't going to be learning any generalizable patterns on the test data set, which is why we focus our data augmentations on the training data set. And I've just readjusted that. I don't want to do that. Beautiful. So we've got a transform ready. Now let's load some data using those transforms. So we'll create train and test data sets and data loaders with data augmentation. So we've done this before. You might want to try it out on your own. So
24:20:12
pause the video if you'd like to test it out. Create a data set and a data loader using these transforms here. And recall that our data set is going to be creating a data set from pizza, steak and sushi for the train and test folders. And that our data loader is going to be batchifying our data set. So let's turn our image folders into data sets. Data sets, beautiful. And I'm going to write here train data augmented just so we know that it's it's been augmented. We've got a few
24:20:48
of similar variable names throughout this notebook. So I just want to be as clear as possible. And I'm going to use, I'll just re import torch vision data sets. We've seen this before, the image folder. So rather than our use our own custom class, we're going to use the existing image folder class that's within torch vision data sets. And we have to pass in here a root. So I'll just get the doc string there, root, which is going to be equal to our trainer, which recall is the path
24:21:19
to our training directory. Got that saved. And then I'm going to pass in here, the transform is going to be train transform trivial. So our training data is going to be augmented. Thanks to this transform here, and transforms trivial augment wide. You know where you can find more about trivial augment wide, of course, in the pie torch documentation, or just searching transforms trivial augment wide. And did I spell this wrong? trivial. Oh, train train transform. I spelled
24:21:59
that wrong. Of course I did. So test data, let's create this as test data simple, equals data sets dot image folder. And the root D is going to be here the test directory. And the transform is just going to be what the test transform simple. Beautiful. So now let's turn these data sets into data loaders. So turn our data sets into data loaders. We're going to import os, I'm going to set the batch size here to equal to 32. The number of workers that are going to
24:22:41
load our data loaders, I'm going to set this to os dot CPU count. So there'll be one worker per CPU on our machine. I'm going to set here the torch manual seed to 42, because we're going to shuffle our training data. Train data loader, I'm going to call this augmented equals data loader. Now I just want to I don't need to re import this, but I just want to show you again from torch dot utils. You can never have enough practice right dot data. Let's import data loader.
24:23:16
So that's where we got the data loader class from. Now let's go train data augmented. We'll pass in that as the data set. And I'll just put in here the parameter name for completeness. That's our data set. And then we want to set the batch size, which is equal to batch size. I'm going to set shuffle equal to true. And I'm going to set num workers equal to num workers.
24:23:45
Beautiful. And now let's do that again with the test data loader that this time test data loader. I'm going to call this test data loader simple. We're not using any data augmentation on the test data set, just turning our images, our test images into tenses. The data set here is going to be test data simple. Going to pass in the batch size equal to batch size. So both our data loaders will have a batch size of 32. Going to keep shuffle on false.
24:24:14
And num workers, I'm going to set to num workers. Look at us go. We've already got a data set and a data loader. This time, our data loader is going to be augmented for the training data set. And it's going to be nice and simple for the test data set. So this is really similar, this data loader to the previous one we made. The only difference in this modeling experiment is that we're going to be adding data augmentation, namely trivial augment wide.
24:24:47
So with that being said, we've got a data set, we've got a data loader. In the next video, let's construct and train model one. In fact, you might want to give that a go. So you can use our tiny VGG class to make model one. And then you can use our train function to train a new tiny VGG instance with our training data loader augmented and our test data loader simple. So give that a go and we'll do it together in the next video. I'll see you there.
24:25:17
Now that we've got our data sets and data loaders with data augmentation ready, let's now create another model. So 9.3, we're going to construct and train model one. And this time, I'm just going to write what we're going to doing, going to be doing sorry. This time, we'll be using the same model architecture, but we're changing the data here. Except this time, we've augmented the training data. So we'd like to see how this performs
24:25:56
compared to a model with no data augmentation. So that was our baseline up here. And that's what you'll generally do with your experiments. You'll start as simple as possible and introduce complexity when required. So create model one and send it to the target device, that is, to the target device. And because of our helpful selves previously, we can create a manual seed here, torch.manualseed. And we can create model one, leveraging the class that we created before.
24:26:29
So although we built tiny VGG from scratch in this video, in this section, sorry, in subsequent coding sessions, because we've built it from scratch once and we know that it works, we can just recreate it by calling the class and passing in different variables here. So let's get the number of classes that we have in our train data augmented classes. And we're going to send it to device. And then if we inspect model one, let's have a look. Wonderful. Now let's keep going.
24:27:03
We can also leverage our training function that we did. You might have tried this before. So let's now train our model. She's going to put here. Wonderful. Now we've got a model and data loaders. Let's create what do we have to do? We have to create a loss function and an optimizer and call upon our train function that we created earlier to train and evaluate our model. Beautiful. So I'm going to set the random seeds, torch dot manual seeds, and torch dot CUDA, because we're
24:27:48
going to be using CUDA. Let's set the manual seed here 42. I'm going to set the number of epochs. We're going to keep many of the parameters the same. Set the number of epochs, num epochs equals five. We could of course train this model for longer if we really wanted to by increasing the number of epochs. But now let's set up the loss function. So loss FN equals NN cross entropy loss. Don't forget this just came into mind. Loss function often as well in PyTorch is called
24:28:21
criterion. So the criterion you're trying to reduce. But I just like to call it loss function.
24:28:28
And then we're going to have optimizer. Let's use the same optimizer we use before torch dot opt in dot atom. Recall SGD and atom are two of the most popular optimizers. So model one dot parameters. Then the parameters we're going to optimize. We're going to set the learning rate to zero zero one, which is the default for the atom optimizer in PyTorch. Then we're going to start the timer. So from time it, let's import the default timer as timer. And we'll go start time
24:29:02
equals timer. And then let's go train model one. How can we do this? Well, we're going to get a results dictionary as model one results. We're going to call upon our train function. Inside our train function, we'll pass the model parameter as model one. For the train data loader parameter, we're going to pass in train data loader augmented. So our augmented training data loader. And for the test data loader, we can pass in here test data loader. Simple. Then we can write our
24:29:39
optimizer, which will be the atom optimizer. Our loss function is going to be an n cross entropy loss, what we've created above. And then we can set the number of epochs is going to be equal to num epochs. And then if we really wanted to, we could set the device equal to device, which will be our target device. And now let's end the timer and print out how long it took.
24:30:06
Took n time equals timer. And we'll go print total training time for model one is going to be n time minus start time. And oh, it would help if I could spell, we'll get that to three decimal places. And that'll be seconds. So you're ready? We look how quickly we built a training pipeline for model one. And look how big easily we created it. So go ask for coding all of that stuff up before. Let's train our second model, our first model using data augmentation. You're ready? Three,
24:30:50
two, one, let's go. No errors. Beautiful. We're going nice and quick here.
24:30:59
So oh, about just over seven seconds. So what what GPU do I have currently? Just keep this in mind that I'm using Google Colab Pro. So I get preference in terms of allocating a faster GPU. Your model training time may be longer than what I've got, depending on the GPU. It also may be faster, again, depending on the GPU. But we get about seven seconds, but it looks like our model with data augmentation didn't perform as well as our model without data augmentation.
24:31:29
Hmm. So how long did our model before without data augmentation take the train? Oh, just over seven seconds as well. But we got better results in terms of accuracy on the training and test data sets for model zero. So maybe data augmentation doesn't help in our case. And we kind of hinted at that because the loss here was already going down. We weren't really overfitting yet. So recall that data augmentation is a way to help with overfitting generally. So maybe that wasn't the best step to
24:32:03
try and improve our model. But let's nonetheless keep evaluating our model. In the next video, we're going to plot the loss curves of model one. So in fact, you might want to give that a go. So we've got a function plot loss curves, and we've got some results in a dictionary format. So try that out, plot the loss curves, and see what you see. Let's do it together in the next video. I'll see you there.
24:32:31
In the last video, we did the really exciting thing of training our first model with data augmentation. But we also saw that quantitatively, it looks like that it didn't give us much improvement. So let's keep evaluating our model here. I'm going to make a section. Recall that one of my favorite ways or one of the best ways, not just my favorite, to evaluate the performance of a model over time is to plot the loss curves. So a loss curve helps you evaluate your model's performance
24:33:04
over time. And it will also give you a great visual representation or a visual way to see if your model is underfitting or overfitting. So let's plot the loss curves of model one results and see what happens. We're using this function we created before. And oh my goodness, is that going in the right direction? It looks like our test loss is going up here. Now, is that where we want it to go? Remember the ideal direction for a loss curve is to go down over time because loss is measuring
24:33:38
what? It's measuring how wrong our model is. And the accuracy curve looks like it's all over the place as well. I mean, it's going up kind of, but maybe we don't have enough time to measure these things. So an experiment that you could do is train both of our models model zero and model one for more epochs and see if these loss curves flatten out. So I'll pose you the question, is our model underfitting or overfitting right now or both? So if we want to have a look at the
24:34:09
loss curves, our just right is for the loss that is, this is not for accuracy, this is for loss over time, we want it to go down. So for me, our model is underfitting because our loss could be lower, but it also looks like it's overfitting as well. So it's not doing a very good job because our test loss is far higher than our training loss. So if we go back to section four of the LearnPyTorch.io book, what should an ideal loss curve look like? I'd like you to start thinking of some ways
24:34:43
that we could deal with overfitting of our model. So could we get more data? Could we simplify it? Could we use transfer learning? We're going to see that later on, but you might want to jump ahead and have a look. And if we're dealing with underfitting, what are some other things that we could try with our model? Could we add some more layers, potentially another convolutional block? Could we increase the number of hidden units per layer? So if we've got currently 10 hidden units
24:35:08
per layer, maybe you want to increase that to 64 or something like that? Could we train it for longer? That's probably one of the easiest things to try with our current training functions. We could train for 20 epochs. So have a go at this, reference this, try out some experiments with, see if you can get these loss curves more towards the ideal shape. And in the next video, we're going to keep pushing forward. We're going to compare our model results. So we've done two experiments.
24:35:35
Let's now see them side by side. We've looked at our model results individually, and we know that they could be improved. But a good way to compare all of your experiments is to compare your model's results side by side. So that's what we're going to do in the next video. I'll see you there. Now that we've compared our models loss curves on their own individually, how about we compare our model results to each other? So let's have a look at comparing our model
24:36:05
results. And so I'm going to write a little note here that after evaluating our modeling experiments on their own, it's important to compare them to each other. And there's a few different ways to do this. There's a few different ways to do this. Number one is hard coding. So like we've done, we've written functions, we've written helper functions and whatnot, and manually plotted things. So I'm just going to write in here, this is what we're doing.
24:36:42
Then, of course, there are tools to do this, such as PyTorch plus TensorBoard. So I'll link to this, PyTorch TensorBoard. We're going to see this in a later section of the course. TensorBoard is a great resource for tracking your experiments. If you'd like to jump forward and have a look at what that is in the PyTorch documentation, I'd encourage you to do so. Then another one of my favorite tools is weights and biases. So these are all going to involve some code as well, but they help out
24:37:13
with automatically tracking different experiments. So weights and biases is one of my favorite, and you've got platform for experiments. That's what you'll be looking at. So if you run multiple experiments, you can set up weights and biases pretty easy to track your different model hub parameters. So PyTorch, there we go. Import weights and biases, start a new run on weights and biases. You can save the learning rate value and whatnot, go through your data and just log everything there.
24:37:43
So this is not a course about different tools. We're going to focus on just pure PyTorch, but I thought I'd leave these here anyway, because you're going to come across them eventually, and MLflow is another one of my favorites as well. We have ML tracking, projects, models, registry, all that sort of stuff. If you'd like in to look into more ways to track your experiments, there are some extensions. But for now, we're going to stick
24:38:08
with hard coding. We're just going to do it as simple as possible to begin with. And if we wanted to add other tools later on, we can sure do that. So let's create a data frame for each of our model results. We can do this because our model results recall are in the form of dictionaries. So model zero results. But you can see what we're doing now by hard coding this, it's quite cumbersome. Can you imagine if we had say 10 models or even just five models, we'd have to really
24:38:38
write a fair bit of code here for all of our dictionaries and whatnot, whereas these tools here help you to track everything automatically. So we've got a data frame here. Model zero results over time. These are our number of epochs. We can notice that the training loss starts to go down. The testing loss also starts to go down. And the accuracy on the training and test data set starts to go up. Now, those are the trends that we're looking for. So an experiment you could try would
24:39:06
be to train this model zero for longer to see if it improved. But we're currently just interested in comparing results. So let's set up a plot. I want to plot model zero results and model one results on the same plot. So we'll need a plot for training loss. We'll need a plot for training accuracy, test loss and test accuracy. And then we want two separate lines on each of them. One for model zero and one for model one. And this particular pattern would be similar regardless if
24:39:37
we had 10 different experiments, or if we had 10 different metrics we wanted to compare, you generally want to plot them all against each other to make them visual. And that's what tools such as weights and biases, what TensorBoard, and what ML flow can help you to do. I'm just going to get out of that, clean up our browser. So let's set up a plot here. I'm going to use matplotlib. I'm going to put in a figure. I'm going to make it quite large because we want four
24:40:04
subplots, one for each of the metrics we want to compare across our different models. Now, let's get number of epochs. So epochs is going to be length, or we'll turn it into a range, actually, range of Len model zero DF. So that's going to give us five. Beautiful range between zero and five. Now, let's create a plot for the train loss. We want to compare the train loss across model zero and the train loss across model one. So we can go PLT dot subplot. Let's create a plot with two
24:40:45
rows and two columns. And this is going to be index number one will be the training loss. We'll go PLT dot plot. I'm going to put in here epochs and then model zero DF. Inside here, I'm going to put train loss for our first metric. And then I'm going to label it with model zero. So we're comparing the train loss on each of our modeling experiments. Recall that model zero was our baseline model. And that was tiny VGG without data augmentation. And then we tried out model one,
24:41:19
which was the same model. But all we did was we added a data augmentation transform to our training data. So PLT will go x label. They both used the same test data set and PLT dot legend. Let's see what this looks like. Wonderful. So there's our training loss across two different models. So we notice that model zero is trending in the right way. Model one kind of exploded on epoch number that would be zero, one, two, or one, depending how you're counting. Let's just say epoch number
24:41:56
two, because that's easier. The loss went up. But then it started to go back down. So again, if we continued training these models, we might notice that the overall trend of the loss is going down on the training data set, which is exactly what we'd like. So let's now plot, we'll go the test loss. So I'm going to go test loss here. And then I'm going to change this. I believe if I hold control, or command, maybe, nope, or option on my Mac keyboard,
24:42:25
yeah, so it might be a different key on Windows. But for me, I can press option and I can get a multi cursor here. So I'm just going to come back in here. And that way I can backspace there and just turn this into test loss. Wonderful. So I'm going to put this as test loss as the title. And I need to change the index. So this will be index one, index two, index three, index four. Let's see what this looks like. Do we get the test loss? Beautiful. That's what we get.
24:42:55
However, we noticed that model one is probably overfitting at this stage. So maybe the data augmentation wasn't the best change to make to our model. Recall that even if you make a change to your model, such as preventing overfitting or underfitting, it won't always guarantee that the change takes your model's evaluation metrics in the right direction. Ideally, loss is going from top left to bottom right over time. So looks like model zero is winning out here at the moment
24:43:25
on the loss front. So now let's plot the accuracy for both training and test. So I'm going to change this to train. I'm going to put this as accuracy. And this is going to be index number three on the plot. And do we save it as, yeah, just act? Wonderful. So I'm going to option click here on my Mac. This is going to be train. And this is going to be accuracy here. And then I'll change this one to accuracy. And then I'm going to change this to accuracy. And this is going to be plot number four,
24:44:01
two rows, two columns, index number four. And I'm going to option click here to have two cursors, test, act. And then I'll change this to test, act. And I'm going to get rid of the legend here. It takes a little bit to plot because we're doing four graphs in one hit. Wonderful. So that's comparing our models. But do you see how we could potentially functionalize this to plot, however, many model results that we have? But if we had say another five models, we did another five
24:44:32
experiments, which is actually not too many experiments on a problem, you might find that sometimes you do over a dozen experiments for a single modeling problem, maybe even more. These graphs can get pretty outlandish with all the little lines going through. So that's again what tools like TensorBoard, weights and biases and MLflow will help with. But if we have a look at the accuracy, it seems that both of our models are heading in the right direction.
24:44:59
We want to go from the bottom left up in the case of accuracy. But the test accuracy that's training, oh, excuse me, is this not training accuracy? I messed up that. Did you catch that one?
24:45:12
So training accuracy, we're heading in the right direction, but it looks like model one is yeah, still overfitting. So the results we're getting on the training data set aren't coming over to the testing data set. And that's what we really want our models to shine is on the test data set. So metrics on the training data set are good. But ideally, we want our models to perform well on the test data set data it hasn't seen before.
24:45:38
So that's just something to keep in mind. Whenever you do a series of modeling experiments, it's always good to not only evaluate them individually, evaluate them against each other. So that way you can go back through your experiments, see what worked and what didn't. If you were to ask me what I would do for both of these models, I would probably train them for longer and maybe add some more hidden units to each of the layers and see where the results go from
24:46:02
there. So give that a shot. In the next video, let's see how we can use our trained models to make a prediction on our own custom image of food. So yes, we used a custom data set of pizza steak and sushi images. But what if we had our own, what if we finished this model training and we decided, you know what, this is a good enough model. And then we deployed it to an app like neutrify dot app, which is a food recognition app that I'm personally working on. Then we wanted to
24:46:32
upload an image and have it be classified by our pytorch model. So let's give that a shot, see how we can use our trained model to predict on an image that's not in our training data and not in our testing data. I'll see you in the next video. Welcome back. In the last video, we compared our modeling experiments. Now we're going to move on to one of the most exciting parts of deep learning. And that is making a prediction on a custom image. So although we've trained a model on custom data,
24:47:14
how do you make a prediction on a sample slash image in our case? That's not in either the training or testing data set. So let's say you were building a food recognition app, such as neutrify, take a photo of food and learn about it. You wanted to use computer vision to essentially turn foods into QR codes. So I'll just show you the workflow here. If we were to upload this image of my dad giving two thumbs up for a delicious pizza. And what does neutrify predicted
24:47:48
as pizza? Beautiful. So macaronutrients that you get some nutrition information and then the time taken. So we could replicate a similar process to this using our trained PyTorch model, or be it. It's not going to be too great of results or performance because we've seen that we could improve our models, but based on the accuracy here and based on the loss and whatnot. But let's just see what it's like, the workflow. So the first thing we're going to do is get a custom image.
24:48:16
Now we could upload one here, such as clicking the upload button in Google Colab, choosing an image and then importing it like that. But I'm going to do so programmatically, as you've seen before. So let's write some code in this video to download a custom image. I'm going to do so using requests and like all good cooking shows, I've prepared a custom image for us. So custom image path. But again, you could use this process that we're going to go through with any of your own images
24:48:48
of pizza, steak or sushi. And if you wanted to train your own model on another set of custom data, the workflow will be quite similar. So I'm going to download a photo called pizza dad, which is my dad, two big thumbs up. And so I'm going to download it from github. So this image is on the course github. And let's write some code to download the image. If it doesn't already exist in our Colab instance. So if you wanted to upload a single image, you could click with this button.
24:49:20
Just be aware that like all of our other data, it's going to disappear if Colab disconnects. So that's why I like to write code. So we don't have to re upload it every time. So if not custom image path is file, let's open a request here or open a file going to open up the custom image path with right binary permissions as F short for file. And then when downloading, this is because our image is stored on github. When downloading an image or when downloading
24:49:54
from github in general, you typically want the raw link need to use the raw file link. So let's write a request here equals request dot get. So if we go to the pytorch deep learning repo, then if we go into, I believe it might be extras, not in extras, it's going to be in images, that would make a lot more sense. Wouldn't it Daniel? Let's get O for pizza dad. So if we have a look, this is pytorch deep learning images, O for pizza dad. There's a big version
24:50:26
of the image there. And then if we click download, just going to give us the raw link. Yeah, there we go. So that's the image. Hey dad, how you doing? Is that pizza delicious? It looks like it. Let's see if our model can get this right. What do you think? Will it? So of course, we want our model to predict pizza for this image because it's got a pizza in it. So custom image path, we're going to download that. I've just put in the raw URL above. So notice the raw
24:50:57
github user content. That's from the course github. Then I'm going to go f dot right. So file, write the request content. So the content from the request, in other words, the raw file from github here. Similar workflow for if you were getting another image from somewhere else on the internet and else if it is already downloaded, let's just not download it. So print f custom image path already exists skipping download. And let's see if this works or run the code. So downloading
24:51:36
data o four pizza dad dot jpeg. And if we go into here, we refresh. There we go. Beautiful. So our data or our custom image, sorry, is now in our data folder. So if we click on this, this is inside Google CoLab now. Beautiful. We got a big nice big image there. And there's a nice big pizza there. So we're going to be writing some code over the next few videos to do the exact same process as what we've been doing to import our custom data set for our custom image. What do we still have to
24:52:13
do? We still have to turn it into tenses. And then we have to pass it through our model. So let's see what that looks like over the next few videos. We are up to one of the most exciting parts of building dev learning models. And that is predicting on custom data in our case, a custom image of a photo of my dad eating pizza. So of course, we're training a computer vision model on here on pizza steak and sushi. So hopefully the ideal result for our model to predict on this image
24:52:45
will be pizza. So let's keep going. Let's figure out how we can get our image, our custom image, our singular image into Tensor form, loading in a custom image with pytorch, creating another section here. So I'm just going to write down here, we have to make sure our custom image is in the same format as the data our model was trained on. So namely, that was in Tensor form with data type torch float 32. And then of shape 64 by 64 by three. So we might need to change the shape of our
24:53:29
image. And then we need to make sure that it's on the right device. Command MM, beautiful. So let's see what this looks like. Hey, so if I'm going to import torch vision. Now the package you use to load your data will depend on the domain you're in. So let's open up the torch vision documentation.
24:53:51
We can go to models. That's okay. So if we're working with text, you might want to look in here for some input and output functions, so some loading functions, torch audio, same thing. Torch vision is what we're working with. Let's click into torch vision. Now we want to look into reading and writing images and videos because we want to read in an image, right? We've got a custom image. We want to read it in. So this is part of your extracurricular, by the way, to go
24:54:17
through these for at least 10 minutes each. So spend an hour if you're going through torch vision. You could do the same across these other ones. It will just really help you familiarize yourself with all the functions of PyTorch domain libraries. So we want to look here's some options for video. We're not working with video. Here's some options for images. Now what do we want to do? We want to read in an image. So we've got a few things here. Decode image. Oh, I've skipped over one.
24:54:44
We can write a JPEG if we wanted to. We can encode a PNG. Let's jump into this one. Read image. What does it do? Read the JPEG or PNG into a three-dimensional RGB or grayscale tensor. That is what we want. And then optionally converts the image to the desired format. The values of the output tensor are you int eight. Okay. Beautiful. So let's see what this looks like. Okay. Mode. The read mode used optionally for converting the image. Let's see what we can do
24:55:16
with this. I'm going to copy this in. So I'll write this down. We can read an image into PyTorch using and go with that. So let's see what this looks like in practice. Read in custom image. I can't explain to you how much I love using deep learning models to predict on custom data. So custom image. We're going to call it you int eight because as we read from the documentation here, it reads it in you int eight format. So let's have a look at what that looks like rather than
24:55:52
just talking about it. Torch vision.io. Read image. What's our target image path? Well, we've got custom image path up here. This is why I like to do things programmatically. So if our collab notebook reset, we could just run this cell again, get our custom image and then we've got it here. So custom image you int eight. Let's see what this looks like. Oh, what did we get wrong? Unable to cast Python instance. Oh, does it need to be a
24:56:23
string expected a value type of string or what found POSIX path? So this the path needs to be a string. Okay. If we have a look at our custom image path, what did we get wrong? Oh, we've got a POSIX path. So let's convert this custom image path into a string and see what happens. Look at that.
24:56:47
That's how image in integer form. I wonder if this is plotable. Let's go PLT dot M show custom image you int eight. Maybe we get a dimensionality problem here in valid shape. Okay. Let's some permute it, permute, and we'll go one, two, zero. Is this going to plot? It's a fairly big image. There we go. Two thumbs up. Look at us. So that is the power of torch vision.io. I owe stands for input output. We were just able to read in our custom image. Now, how about we get some metadata
24:57:24
about this? Let's go. We'll print it up here, actually. I'll keep that there because that's fun to plot it. Let's find the shape of our data, the data type. And yeah, we've got it in Tensor format, but it's you int eight right now. So we might have to convert that to float 32. We want to find out its shape. And we need to make sure that if we're predicting on a custom image, the data that we're predicting on the custom image needs to be on the same device as our model.
24:57:52
So let's print out some info. Print. Let's go custom image Tensor. And this is going to be a new line. And then we will go custom image you int eight. Wonderful. And then let's go custom image shape. We will get the shape parameter custom image shape or attribute. Sorry. And then we also want to know the data type custom image data type. But we have a kind of an inkling because the documentation said it would be you int eight, you int eight, and we'll go D type. Let's have a look.
24:58:30
What do we have? So there's our image Tensor. And it's quite a big image. So custom image shape. So what was our model trained on? Our model was trained on images of 64 by 64. So this image encodes a lot more information than what our model was trained on. So we're going to have to change that shape to pass it through our model. And then we've got an image data type here or Tensor data type of torch you int eight. So maybe that's going to be some errors for us later on.
24:59:01
So if you want to go ahead and see if you can resize this Tensor to 64 64 using a torch transform or torch vision transform, I'd encourage you to try that out. And if you know how to change a torch tensor from you int eight to torch float 32, give that a shot as well. So let's try make a prediction on our image in the next video. I'll see you there.
24:59:26
In the last video, we loaded in our own custom image and got two big thumbs up from my dad, and we turned it into a tensor. So we've got a custom image tensor here. It's quite big though, and we looked at a few things of what we have to do before we pass it through our model. So we need to make sure it's in the data type torch float 32, shape 64, 64, 3, and on the right device. So let's make another section here. We'll go 11.2 and we'll call it making a prediction on a
24:59:56
custom image with a pie torch model with a trained pie torch model. And albeit, our models aren't quite the level we would like them at yet. I think it's important just to see what it's like to make a prediction end to end on some custom data, because that's the fun part, right? So try to make a prediction on an image. Now, I want to just highlight something about the importance of different data types and shapes and whatnot and devices, three of the biggest errors in deep learning.
25:00:28
In let's see what happens if we try to predict on you int eight format. So we'll go model one dot eval and with torch dot inference mode. Let's make a prediction. We'll pass it through our model one. We could use model zero if we wanted to here. They're both performing pretty poorly anyway. Let's send it to the device and see what happens. Oh, no. What did we get wrong here? Runtime error input type. Ah, so we've got you int eight. So this is one of our first errors
25:01:04
that we talked about. We need to make sure that our custom data is of the same data type that our model was originally trained on. So we've got torch CUDA float tensor. So we've got an issue here. We've got a you into eight image data or image tensor trying to be predicted on by a model with its data type of torch CUDA float tensor. So let's try fix this by loading the custom image and convert to torch dot float 32. So one of the ways we can do this is we'll just recreate the
25:01:42
custom image tensor. And I'm going to use torch vision dot IO dot read image. We don't have to fully reload our image, but I'm going to do it anyway for completeness and a little bit of practice. And then I'm going to set the type here with the type method to torch float 32. And then let's just see what happens. We'll go custom image. Let's see what this looks like. I wonder if our model will work on this. Let's just try again, we'll bring this up, copy this down to make a
25:02:17
prediction and custom image dot two device. Our image is in torch float 32 now. Let's see what happens. Oh, we get an issue. Oh my goodness, that's a big matrix. Now I have a feeling that that might be because our image, our custom image is of a shape that's far too large. Custom image dot shape. What do we get? Oh my gosh, 4000 and 3,024. And do you notice as well that our values here are between zero and one, whereas our previous images, do we have an image? There we go. That
25:02:54
our model was trained on what between zero and one. So how could we get these values to be between zero and one? Well, one of the ways to do so is by dividing by 255. Now, why would we divide by 255?
25:03:09
Well, because that's a standard image format is to store the image tensor values in values from zero to 255 for red, green and blue color channels. So if we want to scale them, so this is what I meant by zero to 255, if we wanted to scale these values to be between zero and one, we can divide them by 255. Because that is the maximum value that they can be. So let's see what happens if we do that. Okay, we get our image values between zero and one. Can we plot this image? So plt dot m
25:03:47
show, let's plot our custom image. We got a permute it. So it works nicely with mapplotlib. What do we get here?
25:04:00
Beautiful. We get the same image, right? But it's still quite big. Look at that. We've got a pixel height of or image height of almost 4000 pixels and a width of over 3000 pixels. So we need to do some adjustments further on. So let's keep going. We've got custom image to device. We've got an error here. So this is a shape error. So what can we do to transform our image shape? And you might have already tried this. Well, let's create a transform pipeline to transform our image shape.
25:04:29
So create transform pipeline or composition to resize the image. Because remember, what are we trying to do? We're trying to get our model to predict on the same type of data it was trained on. So let's go custom image transform is transforms dot compose. And we're just going to, since our image is already of a tensor, let's do transforms dot resize, and we'll set the size to the same shape that our model was trained on, or the same size that is. So let's go from torch vision. We don't
25:05:06
have to rewrite this. It's already imported. But I just want to highlight that we're using the transforms package. We'll run that. There we go. We've got a transform pipeline. Now let's see what happens when we transform our target image, transform target image. What happens? Custom image transformed. I love printing the inputs and outputs of our different pipelines here. So let's pass our custom image that we've just imported. So custom image transform, our custom image is recall
25:05:39
of shape. Quite large. We're going to pass it through our transformation pipeline. And let's print out the shapes. Let's go original shape. And then we'll go custom image dot shape. And then we'll go print transformed shape is going to be custom image underscore transformed dot shape. Let's see the transformation. Oh, would you look at that? How good we've gone from quite a large image to a transformed image here. So it's going to be squished and squashed a little. So that's what
25:06:24
happens. Let's see what happens when we plot our transformed image. We've gone from 4000 pixels on the height to 64. And we've gone from 3000 pixels on the height to 64. So this is what our model is going to see. Let's go custom image transformed. And we're going to permute it to be 120.
25:06:47
Okay, so quite pixelated. Do you see how this might affect the accuracy of our model? Because we've gone from custom image, is this going to, oh, yeah, we need to plot dot image. So we've gone from this high definition image to an image that's of far lower quality here. And I can kind of see myself that this is still a pizza, but I know that it's a pizza. So just keep this in mind going forward is that another way that we could potentially improve our model's
25:07:15
performance if we increased the size of the training image data. So instead of 64 64, we might want to upgrade our models capability to deal with images that are of 224 224. So if we have a look at what this looks like 224 224. Wow, that looks a lot better than 64 64. So that's something that you might want to try out later on. But we're going to stick in line with the CNN explainer model.
25:07:47
How about we try to make another prediction? So since we transformed our
25:07:53
image to be the same size as the data our model was trained on. So with torch inference mode, let's go custom image pred equals model one on custom image underscore transformed. Does it work now? Oh my goodness, still not working expected all tensors on the same device. Of course, that's what we forgot here. Let's go to device. Or actually, let's leave that error there. And we'll just copy this code down here. And let's put this custom image transform back on the right
25:08:27
device and see if we finally get a prediction to happen with our model. Oh, we still get an error. Oh my goodness, what's going on here? Oh, we need to add a batch size to it. So I'm just gonna write up here. This will error. No batch size. And this will error. Image not on right device. And then let's try again, we need to add a batch size to our image. So if we look at custom image transformed
25:09:08
dot shape, recall that our images that passed through our model had a batch dimension. So this is another place where we get shape mismatch issues is if our model, because what's going on in neural network is a lot of tensor manipulation. If the dimensions don't line up, we want to perform matrix multiplication and the rules. If we don't play to the rules, the matrix multiplication will fail. So let's fix this by adding a batch dimension. So we can do this by going a custom image transformed.
25:09:43
Let's unsqueeze it on the first dimension and then check the shape. There we go. We add a single batch. So that's what we want to do when we make a prediction on a single custom image. We want to pass it to our model as an image or a batch of one sample. So let's finally see if this will work.
25:10:03
Let's just not comment what we'll do. This, or maybe we'll try anyway, this should work.
25:10:11
Added a batch size. So do you see the steps we've been through so far? And we're just going to unsqueeze this. Unsqueeze on the zero dimension to add a batch size. Oh, it didn't error. Oh my goodness. It didn't error. Have a look at that. Yes, that's what we want. We get a prediction load it because the raw outputs of our model, we get a load it value for each of our custom classes. So this could be pizza. This could be steak. And this could be sushi, depending on the order of
25:10:44
our classes. Let's just have a look. Class to IDX. Did we not get that? Class names.
25:10:56
Beautiful. So pizza steak sushi. We've still got a ways to go to convert this into that. But I just want to highlight what we've done. So note, to make a prediction on a custom image, we had to. And this is something you'll have to keep in mind for almost all of your custom data. It needs to be formatted in the same way that your model was trained on. So we had to load the image and turn it into a tensor. We had to make sure the image was the same data type as the model.
25:11:35
So that was torch float 32. And then we had to make sure the image was the same shape as the data the model was trained on, which was 64, 64, three with a batch size. So that was one, three, 64, 64. And excuse me, this should actually be the other way around. This should be color channels first, because we're dealing with pie torch here. 64. And then finally, we had to make sure the image was on the same device as our model. So they are three of the big ones that we've
25:12:21
talked about so much the same data type or data type mismatch will result in a bunch of issues. Shape mismatch will result in a bunch of issues. And device mismatch will also result in a bunch of issues. If you want these to be highlighted, they are in the learn pie torch.io resource. We have putting things together. Where do we have it? Oh, yeah, no, it's in the main takeaway section, sorry, predicting on your own custom data with a trained model as possible, as long as you format
25:12:53
the data into a similar format to what the model was trained on. So make sure you take care of the three big pie torch and deep learning errors. Wrong data types, wrong data shapes, and wrong devices, regardless of whether that's images or audio or text, these three will follow you around. So just keep them in mind. But now we've got some code to predict on custom images, but it's kind of all over the place. We've got about 10 coding cells here just to make a prediction on a custom
25:13:22
image. How about we functionize this and see if it works on our pizza dad image. I'll see you in the next video. Welcome back. We're now well on our way to making custom predictions on our own custom image data. Let's keep pushing forward. In the last video, we finished off getting some raw model logits. So the raw outputs from our model. Now, let's see how we can convert these logits into prediction labels. Let's write some code. So convert logits to prediction labels. Or let's go
25:14:00
convert logits. Let's first convert them to prediction probabilities. Probabilities. So how do we do that? Let's go custom image pred probes equals torch dot softmax to convert our custom image pred across the first dimension. So the first dimension of this tensor will be the inner brackets, of course. So just this little section here. Let's see what these look like. This will be prediction probabilities. Wonderful. So you'll notice that these are quite
25:14:36
spread out. Now, this is not ideal. Ideally, we'd like our model to assign a fairly large prediction probability to the target class, the right target class that is. However, since our model when we trained it isn't actually performing that all that well. The prediction probabilities are quite spread out across all of the classes. But nonetheless, we're just highlighting what it's like to predict on custom data. So now let's convert the prediction probabilities
25:15:03
to prediction labels. Now, you'll notice that we used softmax because why we are working with multi class classification data. And so we can get the custom image pred labels, the integers, by taking the argmax of the prediction probabilities, custom image pred probes across the first dimension as well. So let's go custom image pred labels. Let's see what they look like. Zero. So the index here with the highest value is index number zero. And you'll notice that it's
25:15:42
still on the coded device. So what would happen if we try to index on our class names with the custom image pred labels? Or maybe that doesn't need to be a plural. Oh, there we go. We get pizza. But you might also have to change this to the CPU later on. Otherwise, you might run into some errors. So just be aware of that. So you notice how we just put it to the CPU. So we get pizza. We got a correct prediction. But this is as good as guessing in my opinion, because these are kind
25:16:16
of spread out. Ideally, this value would be higher, maybe something like 0.8 or above for our pizza dad image. But nonetheless, our model is getting two thumbs up even on this 64 by 64 image. But that's a lot of code that we've written. Let's functionize it. So we can just pass in a file path and get a custom prediction from it. So putting custom image prediction together. Let's go building a function. So we want the ideal outcome is, let's plot our image as well.
25:16:52
Ideal outcome is a function where we plot or where we pass an image path to and have our model predict on that image and plot the image plus the prediction. So this is our ideal outcome. And I think I'm going to issue this as a challenge. So give that a go, put all of our code above together. And you'll just have to import the image, you'll have to process it and whatnot. I know I said we were going to build a function in this video, but we're going to say that to the next video. I'd like
25:17:31
you to give that a go. So start from way back up here, import the image via torture vision.io read image, format it using what we've done, change the data type, change the shape, change the device, and then plot the image with its prediction as the title. So give that a go and we'll do it together in the next video. How'd you go? I just realized I had a typo in the previous cell, but that's all right. Did you give it a shot? Did you put together the custom image prediction
25:18:07
in a function format? I'd love it if you did. But if not, that's okay. Let's keep going. Let's see what that might look like. And there are many different ways that you could do this. But here's one of the ways that I've thought of. So we want to function that's going to pred and plot a target image. We wanted to take in a torch model. And so that's going to be ideally a trained model. We wanted to also take in an image path, which will be of a string. It can
25:18:33
take in a class names list so that we can index it and get the prediction label in string format. So let's put this as a list of strings. And by default, this can equal none. Just in case we just wanted the prediction, it wants to take in a transform so that we can pass it in some form of transform to transform the image. And then it's going to take in a device, which will be by default the target device. So let's write a little doc string here, makes a prediction on a target image
25:19:05
with a trained model and plots the image and prediction. Beautiful. Now what do we have to do first? Let's load in the image. Load in the image just like we did before with torch vision. So target image equals torch vision.io dot read image. And we'll go string on the image path, which will be the image path here. And we convert it to a string just in case it doesn't get passed in as a string. And then let's change it into type torch float 32. Because we want to make sure that
25:19:49
our custom image or our custom data is in the same type as what we trained our model on. So now let's divide the image pixel values by 255 to get them between zero or to get them between zero one as a range. So we can just do this by target image equals target image divided by 255. And we could also just do this in one step up here 255. But I've just put it out there just to let you know that, hey, read image imports image data as between zero and 255. So our model prefers numbers
25:20:30
between zero and one. So let's just scale it there. Now we want to transform our data if necessary. In our case, it is, but it won't always be. So we want this function to be pretty generic predomplot image. So if the transform exists, let's set the target image to the transform, or we'll pass it through the transform that is wonderful. And the transform we're going to get from here. Now what's left to do? Well, let's make sure the model is on the target device.
25:21:05
It might be by default, but if we're passing in a device parameter, we may as well make sure the model is there too. And now we can make a prediction. So let's turn on a vowel slash inference mode and make a prediction with our model. So model, we call a vowel mode, and then with torch dot inference mode, because we're making a prediction, we want to turn our model into inference mode, or put it in inference mode context. Let's add an extra dimension to the image. Let's go target
25:21:39
image. We could do this step above, actually, but we're just going to do it here. From kind of remembering things on the fly here of what we need to do, we're adding a, this is, let's write this down, this is the batch dimension. e g our model will predict on batches of one x image. So we're just unsqueezing it to add an extra dimension at the zero dimension space, just like we did in a previous video. Now let's make a prediction
25:22:09
on the image with an extra dimension. Otherwise, if we don't have that extra dimension, we saw that we get a shape issue. So right down here, target image pred. And remember, this is going to be the raw model outputs, raw logit outputs. We're going to target image pred. And yeah, I believe that's all we need for the prediction. Oh wait, there was one more thing, two device. Me too. Also make sure the target image is on the right device. Beautiful. So fair
25:22:44
few steps here, but nothing we can't handle. All we're really doing is replicating what we've done for batches of images. But we want to make sure that if someone passed any image to our pred and plot image function, that we've got functionality in here to handle that image. And do we get this? Oh, we want just target image to device. Did you catch that error? So let's keep going. Now let's convert the logits. Our models raw logits. Let's convert those
25:23:16
to prediction probabilities. This is so exciting. We're getting so close to making a function to predict on custom data. So we'll set this to target image pred probes, which is going to be torch dot softmax. And we will pass in the target image pred here. We want to get the softmax of the first dimension. Now let's convert our prediction probabilities, which is what we get in the line above. We want to convert those to prediction labels. So let's get the target image pred labels
25:23:49
labels equals torch dot argmax. We want to get the argmax of, or in other words, the index, which is the maximum value from the pred probes of the first dimension as well. Now what should we return here? Well, we don't really need to return anything. We want to create a plot. So let's plot the image alongside the prediction and prediction probability. Beautiful. So plot dot in show, what are we going to pass in here? We're going to pass in here our target image. Now we have to
25:24:27
squeeze this, I believe, because we've added an extra dimension up here. So we'll squeeze it to remove that batch size. And then we still have to permute it because map plot lib likes images in the format color channels last one, two, zero. So remove batch dimension.
25:24:50
And rearrange shape to be hc hwc. That is color channels last. Now if the class names parameter exists, so we've passed in a list of class names, this function is really just replicating everything we've done in the past 10 cells, by the way. So right back up here, we're replicating all of this stuff in one function. So pretty large function, but once we've written it, we can pass in our images as much as we like. So if class names exist, let's set the title
25:25:22
to our showcase that class name. So the pred is going to be class names. Let's index on that pred image, or target image pred label. And this is where we'll have to put it to the CPU, because if we're using a title with map plot lib, map plot lib cannot handle things that are on the GPU. This is why we have to put it to the CPU. And then I believe that should be enough for that. Let's add a little line in here, so that we can have it. Oh, I've missed something.
25:25:56
An outside bracket there. Wonderful. Let's add the prediction probability, because that's always fun to see. So we want target image pred probs. And we want to get the maximum pred problem from that. And we'll also put that on the CPU. And I think we might get this three decimal places. Now this is saying, oh, pred labels, we don't need that. We need just non plural, beautiful. Now, if the class names doesn't exist, let's just set the title equal to f f string, we'll go pred,
25:26:34
target image pred label. Is Google Colab still telling me this is wrong? Target image pred label. Oh, no, we've still got the same thing. It just hasn't caught up with me, and I'm coding a bit fast here. And then we'll pass in the prob, which will be just the same as above. I could even copy this in. Beautiful. And let's now set the title to the title. And we and we will turn the axes off. PLT axes false. Fair bit of code there. But this is going to be a
25:27:11
super exciting moment. Let's see what this looks like. When we pass it in a target image and a target model, some class names, and a transform. Are you ready? We've got our transform ready, by the way, it's back up here. Custom image transform. It's just going to resize our image. So let's see. Oh, this file was updated remotely or in another tab. Sometimes this happens, and usually Google Colab sorts itself out, but that's all right. It doesn't affect our code for now.
25:27:39
Pred on our custom image. Are you ready? Save failed. Would you like to override? Yes, I would. So you might see that in Google Colab. Usually it fixes itself. There we go. Save successfully. Pred and plot image. I was going to say, Google Colab, don't fail me now. We're about to predict on our own custom data. Using a model trained on our own custom data. Image part. Let's pass in custom image path, which is going to be the path to our pizza dad image. Let's go class names,
25:28:13
equals class names, which is pizza, steak, and sushi. We'll pass in our transform to convert our image to the right shape and size custom image transform. And then finally, the target device is going to be device. Are you ready? Let's make a prediction on custom data. One of my favorite things. One of the most fun things to do when building deep learning models. Three, two, one. How did it go? Oh, no. What did we get wrong? CPU. Okay. Such a so close, but yet so far.
25:28:50
Has no attribute CPU. Oh, maybe we need to put this to CPU. That's where I got the square bracket wrong. So that's what we needed to change. We needed to because this is going to be potentially on the GPU. Tag image pred label. We need to put it on the CPU. We need to do that. Why? Because this is going to be the title of our map plot lib plot. And map plot lib doesn't interface too well with data on a GPU. Let's try it again. Three, two, one, running. Oh, look at that.
25:29:25
Prediction on a custom image. And it gets it right. Two thumbs up. I didn't plan this. Our model is performing actually quite poorly. So this is as good as a guess to me. You might want to try this on your own image. And in fact, if you do, please share it with me. I would love to see it. But you could potentially try this with another model. See what happens? Steak. Okay, there we go. So even though model one performs worse quantitatively, it performs better qualitatively. So that's the
25:29:55
power of a visualize, visualize, visualize. And if we use model zero, also, which isn't performing too well, it gets it wrong with a prediction probability of 0.368, which isn't too high either. So we've talked about a couple of different ways to improve our models. Now we've even got a way to make predictions on our own custom images. So give that a shot. I'd love to see your custom predictions, upload an image here if you want, or download it into Google Colab using
25:30:25
code that we've used before. But we've come a fairly long way. I feel like we've covered enough for custom data sets. Let's summarize what we've covered in the next video. And I've got a bunch of exercises and extra curriculum for you. So this is exciting stuff. I'll see you in the next video.
25:30:47
In the last video, we did the very exciting thing of making a prediction on our own custom image, although it's quite pixelated. And although our models performance quantitatively didn't turn out to be too good qualitatively, it happened to work out. But of course, there are a fair few ways that we could improve our models performance. But the main takeaway here is that we had to do a bunch of pre processing to make sure our custom image was in the same format as what our model
25:31:16
expected. And this is quite a lot of what I do behind the scenes for Nutrify. If you upload an image here, it gets pre processed in a similar way to go through our image classification model to output a label like this. So let's get out of this. To summarize, I've got a colorful slide here, but we've already covered this predicting on custom data. These are three things to make sure of, regardless of whether you're using images, text or audio, make sure your data is in the right
25:31:46
data type. In our case, it was torch float 32. Make sure your data is on the same device as the model. So we had to put our custom image to the GPU, which was where our model also lived. And then we had to make sure our data was in the correct shape. So the original shape was 64, 64, 3. Actually, this should be reversed, because it was color channels first. But the same principle remains here. We had to add a batch dimension and rearrange if we needed. So in our case, we used images of this
25:32:17
shape batches first color channels first height width. But depending on your problem will depend on your shape, depending on the device you're using will depend on where your data and your model lives. And depending on the data type you're using will depend on what you're using for torch float 32 or something else. So let's summarize. If we go here main takeaways, you can read through these, but some of the big ones are pie torch has many built in functions to deal with all kinds
25:32:49
of data from vision to text to audio to recommendation systems. So if we look at the pie torch docs,
25:32:57
you're going to become very familiar with these over time. We've got torch audio data, torch text, torch vision is what we practiced with. And we've got a whole bunch of things here for transforming and augmenting images, data sets, utilities, operators, and torch data is currently in beta. But this is just something to be aware of later on. So it's a prototype library right now, but by the time you watch this, it might be available. But it's another way of loading data.
25:33:24
So just be aware of this for later on. And if we come back to up here, if applied to watch built in data loading functions, don't suit your requirements, you can write your own custom data set classes by subclassing torch dot utils dot data dot data set. And we saw that way back up here in option number two. Option two, here we go, loading image data with a custom data set, wrote plenty of code to do that. And then a lot of machine learning is dealing with the
25:33:56
balance between overfitting and underfitting. We've got a whole section in the book here to check out what an ideal loss curve should look like and how to deal with overfitting, how to deal with underfitting. It's it is a fine line. So much of the research and machine learning is actually dedicated towards this balance. And then three big things for being aware of when you're predicting on your own custom data, wrong data types, wrong data shapes,
25:34:21
and wrong devices. This will follow you around, as I said, and we saw that in practice to get our own custom image ready for a trained model. Now, we have some exercises here. If you'd like the link to it, you can go to loan pytorch.io section number four exercises, and of course, extra curriculum. A lot of the things I've mentioned throughout the course that would be a good resource to check out contained in here. But the exercises, this is this is your time to shine,
25:34:47
your time to practice. Let's go back to this notebook, scroll right down to the bottom. Look how much code we've written. Goodness me, exercises for all exercises and extra curriculum.
25:35:02
See here, turn that into markdown. Wonderful. And so if we go in here, you've got a couple of resources. There's an exercise template notebook for number four, and example solutions for notebook number four, which is what we're working on now. So of course, I'd encourage you to go through the pytorch custom data sets exercises template first. Try to fill out all of the code here on your own. So we've got some questions here. We've got some dummy code. We've got some comments.
25:35:32
So give that a go. Go through this. Use this book resource to reference. Use all the code we've written. Use the documentation, whatever you want. But try to go through this on your own. And then if you get stuck somewhere, you can look at an example solution that I created, which is here, pytorch custom data sets exercise solutions. And just be aware that this is just one way of doing things. It's not necessarily the best. It's just a way to reference what
25:35:57
you're writing to what I would do. And there's actually now live walkthroughs of the solutions, errors and all on YouTube. So if you go to this video, which is going to mute. So this is me live streaming the whole thing, writing a bunch of pytorch code. If you just keep going through all of that, you'll see me writing all of the solutions, running into errors, trying different things, et cetera, et cetera. But that's on YouTube. You can check that out on your own time. But I feel
25:36:25
like we've covered enough exercises. Oh, by the way, this is in the extras exercises tab of the pytorch deep learning repo. So extras exercises and solutions that are contained in there.
25:36:39
Far out. We've covered a lot. Look at all that. So that has been pytorch custom data sets. I will see you in the next section. Holy smokes. That was a lot of pytorch code.
25:36:56
But if you're still hungry for more, there is five more chapters available at learnpytorch.io, which cover transfer learning, my favorite topic, pytorch model experiment tracking, pytorch paper replicating, and pytorch model deployment. How do you get your model into the hands of others? And if you'd like to learn in this video style, the videos for those chapters are available at zero to mastery.io. But otherwise, happy machine learning. And I'll see you next time.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat