Skip to main content
Loading Explore guides…
Convolutional Neural Networks · Acedra
Home
Explore
Computer Science
Convolutional Neural Networks
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
249 segments
Copy
Hide video
0:04
Okay it's that time, so let's go ahead and get started on what we will be calling lecture eight. So you'll have noticed that there was no lecture seven this year, we had the holiday, so our last lecture was lecture six but the naming convention that we were using is by week rather than by how many lectures we've had before, so this is lecture eight. So we hope as usual that you will participate in things like the Discourse and so just look for that “Lecture 8”
0:32
category for this one. Okay, so last time that we were together for a lecture, we talked about neural networks: we defined neural networks, we talked about all kinds of different things like how to represent them both from a mathematical standpoint like notation but also in this function graph that we had talked about. And in particular, we really focused on two layer neural networks. So
1:01
in particular, one layer was creating a new set of features and that was a new idea for us last lecture, this idea of learning a set of features as part of the learning that we were doing rather than just having them be something that we specify in advance, that we come up with in advance. So that was a new idea and when we were doing that, we were having our layer that was learning those features be fully connected, so every output was connected to every input by a set of weights
1:31
and then finally we had a fully connected layer that did the classification or did the regression. In some sense, that wasn't new, that layer, like you could think of the neural networks that we talked about last time with two layers. If you got rid of that hidden layer, you're back to logistic regression or linear regression or one of the things we've seen before. So in some sense, at this point, we've talked about one layer neural networks, we've talked about two-layer
1:53
neural networks, and today we're going to talk about deep neural networks which basically means more than two layers and, in particular, more than one hidden layer. So last time, we had this output layer that did the classification, put the features we've learned together into either a classification or a regression, and then we had the hidden layer that was constructing the features and so that's what we're going to be growing: we're going to
2:15
be growing the number of hidden layers. So we'll talk about that today. In particular, there are lots of ways that you could do that, but we're going to talk about one particular way and hopefully in a moment you'll see why. It's been a very popular way to have a deep neural network. This is convolutional neural networks, it might be called CNN, might be called ConvNets, there are lots of different words but we're going to talk about that as another hypothesis
2:42
class for things like classification, things like regression. Along the way, we're going to describe the layers that really define convolutional networks, so things like filters and max pooling which we'll have seen from the reading and time permitting we'll talk about how that learning is a little bit different than what you had seen in fully connected neural networks, although we'll still be doing these ideas of gradient descent, stochastic gradient descent
3:07
and then the way that you specifically apply those in neural networks like back propagation. So it's not so different, it's just that the back propagation looks a little bit different. Okay, so let's go ahead and start talking about this, but let's also start talking about why are we talking about convolutional neural networks as a particular example of deep learning and the reality is that there's been this big change, this big excitement, around machine learning,
3:36
in general, and deep neural nets, in particular, for the past few years and a lot of that, in some ways, really kicked off with convolutional neural nets, although I'll add a little nuance to that statement in a moment. Okay, so there was this ImageNet large scale visual recognition challenge. Basically the idea is that there were tons of images that have been labeled with things like Amazon Mechanical Turk, they're in this particular challenge. They restricted to about a thousand
4:02
categories, these are like the classes we've been talking about in classification. The exact setup is a little bit more complex than what I've been describing. If you're really interested in this, I strongly recommend you check out the Wikipedia article is pretty cool. Also, there's this great
4:17
retrospective by Olga Russakovsky and a number of other people with Fei-Fei Li and I've cited that at the bottom of this slide as well, so those are pretty cool to check out, but anyway there's this challenge with a thousand categories—so that's a lot more classes than we've typically been dealing with—and they're all these different image categories, like “is there a strawberry,” “is there a cow,” “what's going on in this image?” And the idea was to label the images correctly.
4:44
That was the challenge given to the various teams and so if you look at how, in this case, four teams performed in 2011 (so 2010 was the first year of the challenge), you can see they're all getting over 20% of the images wrong. So that's the way you should be reading this vertical axis: it goes from 0 to 1, and 1 would mean you're basically getting everything wrong.
5:08
And so in 2010 and 2011, the winners of this challenge were using things that we haven't really talked about so much in this class but were really big at the time called support vector machines
5:18
and then something really different happened in 2012 where this one particular submission was, in fact, a deep convolutional neural network and it was called AlexNet. This was a convolutional neural network around eight layers. We'll see what that means shortly, it's also, I have to say, about eight layers because the way people count layers can be different from one thing to another, but you get a sense of how many there were: there more than two but not yet a hundred.
5:50
And it just really outperformed everything else, so much so that by the next year, basically everybody, certainly most of the entries, were convolutional neural networks and, in fact, in 2015, the ImageNet winner was a much deeper convolutional neural network having over 100 layers by Microsoft, and so this seems like it was really changing things. I think this paper has gotten like 60,000 citations or something, it's probably even
6:16
more now. I feel like these types of papers you blink and it's like a thousand more citations.
6:22
But it's also worth noting that, like many things in science, it really wasn't as much of a phase change as it looks like from the outside. A lot of these tools had been building up for years.
6:34
So while there's this very recent AI boom, which was very much set off by AlexNet and related work,
6:42
in the 60s and 80s and today, neural networks were around and they were very popular. And so the idea of a neural network didn't start existing and, in fact, the idea of a convolutional neural network didn't start existing. In 1980, there was something like a ConvNet, this neocognitron. In 1989, Yann LeCun et al were looking at ConvNets for classifying images, they were recognizing digits for post office use, they were using back propagation
7:10
and so you can ask yourself: what changed from the 80s and 90s? What really set this off? And I think there are quite a few different things. So one is there's just better computation, in general, and that's been the story of a lot of the past few decades. GPUs really made a huge difference, they were perfectly aligned with the computations that people were doing in convolutional neural nets,
7:33
they really massively enhanced parallelization. And then also, there was just a lot more labeled data, so there's things like Amazon Mechanical Turk, there's lots of data on the Internet, for better or for worse. So this is an area of controversy sometimes that there's so many images that people share and they aren't necessarily realizing that they're sharing it publicly and those are getting used and swept up or maybe even if they're not sharing them publicly but there's
7:57
certainly a lot more labeled images now and that can really lend itself to this kind of thing.
8:05
Okay so this is just a little teaser for “why convolutional neural nets?” They're really a big part of this AI boom that we've been seeing in the past few years and a really key component of that story. But of course, we also care about them because they can do useful things. So there are a lot of great potential uses of image classification: the ability to detect tumors from medical scans, certainly that's something we would love to do, we'd love to detect different
8:32
types of tumors there. I mean, in general, there are so many things that we'd love to do with medical imaging and this seems to have a lot of possibilities here. Also, of course, image searches online, we really engage with images. And autonomous driving: we want to be able to detect things like stop signs and do some pretty complex things with images and so for all of these reasons we'd really like the ability to work with these kinds of data structures and certainly we haven't
8:59
looked at anything that's a very structured data set yet in this class, we've looked at individual data points. Okay so we're gonna be talking a lot about images today and so let's just recall that images are made of pixels. So one of the big questions we'll be starting off with is: how do I encode this image in a way that now I can do learning on it in the way that we're used to in this class? We have to have a data point that's like an x_1, x_2, x_3, x_4 and we have to
9:26
put it in and put it into our classifier and so this is starting to get us to the point where we see “oh, where are those x's going to be coming from?” They're going to be coming from the pixels.
9:35
Okay, so in particular today, we're going to focus on particularly simple form of images: images that are gray scale images. And in fact, we're going to go even simpler than that: we're going to look at black and white images just for today. You can totally use other types of images, other types of data with convolutional neural nets. This will basically be purely for illustration. So for instance, if I had this image “hi”,
10:00
it might be composed of a bunch of pixels. Each pixel, in general, when we're looking at grayscale images is going to take a value between 0 and P. Again, in today's particularly simple examples, we're going to have just 0 being black and 1 being white and so P is very simple: it's just 1. Even for grayscale images, you don't have to do that in general. So in particular, in your lab you're going to see P being a more typical grayscale image value of 255.
10:30
Okay but here, we're gonna be focusing on just these these 0-1 images again, just for ease of illustration. And so if I had this image that was the word “hi,” I would encode it in zeros and ones in the following way. Now again, we still want to ask: okay, well how are we going to actually use this image as an input for a neural network? Because again, we think about what have we been doing to our data to turn it into an input to some
11:00
kind of machine learning model? Well again, we've had something like x_1, x_2, x_3, x_4 and so what we can do is we can take this image and essentially flatten it and that can turn it into the x_1, x_2, x_3, x_4 and so what I mean by that is exactly the following:
11:15
I just find some ordering of the pixels and I call the first one x_1, and I call the second one x_2, and I call the third one x_3. In some senses, just in terms of encoding this, the ordering isn't so important. In other senses, it will be important and we'll get to that shortly.
11:34
Okay so at this point, I have a vector of values: I have my x_1, I have my x_2, I have my x_3, up to my x_n, or... I always, always get this wrong: they should be superscripts. Oh no, no, no, they shouldn't be, they should be subscripts, so these are in fact the dimensions of one data point. Great okay, so these are all my features essentially for this data point, for this image and so now I want to feed it into a neural net and so let's first ask:
12:02
what would that look like with the neural nets that we were looking at before?
12:06
Okay so previously, we saw these two layer neural nets in this class
12:14
and so this is the example we saw from last week. So one thing that we've done here is we put these parentheses around the superscripts just to super emphasize that they were superscripts. You'll have noticed in the reading and the labs and everything that eventually that just gets to be too much to write and so we just write the superscripts and so that's always a little tricky making sure when it's a superscript and when it's a power
12:35
but here, we're just emphasizing that. We have these two layers and so now we want to say, “okay, how would we feed an image into this?” Well we totally can. I mean we have our x_1 through our number of features that we feed in the beginning and so we already have the ability to put this into a two-layer neural network, to put this image in and do something with it. The only issue is that we could do better essentially, that we could do things to have better performance
13:05
and that's essentially what we're going to be focusing on. So something I want to highlight here is, remember: we had these two layers and the first layer was constructing a set of features
13:15
and then the second layer, this output layer, was taking those features and then making some kind of classification. And so what we're going to be focusing on today is we're not going to change anything about the output layer. We're still going to take features, do some kind of classification with it or regression, and that will be unchanged, that will look like something like maybe logistic regression or linear regression or what have you. What we're going to be focusing on
13:41
is that hidden input layer or the, in general, the build up of the feature. Okay so something to notice about all the layers that we talked about last week is that they're all fully connected. So every input is connected to every output by weight. Again, we're not going to change anything about that in the output layer: we're still going to have our usual logistic regression or linear regression, but we are going to change that in the hidden layers.
14:10
So instead of having this fully connected situation where every input is connected to every output, we're going to think about, essentially, is there anything more that we know about this problem if we're looking at something like images? And in fact there is. And in general, a rule of thumb in machine learning is at least to get started if you know something about the problem, you don't want to force your method to try to relearn it, you want to encode that,
14:33
you want to add that into the method so it can focus on learning the things that are important.
14:37
And so what do we know about an image problem that we didn't know maybe perhaps about a general problem? Well there's a lot of things that are very special about images. So one is spatial locality. Like let's say that I'm looking at some kind of self-driving car application and I want to detect stop signs, I want to know is there a stop sign in this picture. Well, in general, I expect the stop sign is going to be in a bunch of pixels that are near
15:01
each other, I don't expect that the stop sign is going to be like up in the upper right hand corner and also in the lower left hand corner and like dispersed throughout the image. I think that the way that stop signs work is they tend to be in the same place, all their parts are in the same place. So that's something that we have about images that we would like to encode. Another thing we
15:20
have about images or that we know about images is translation and variants. So I could have a stop sign over on this part of the image, but I could also have a stop sign in the middle of the image, I could also have a stop sign in another part of the image and those are all stop signs and I would not want to have a situation where I need to have my neural network learn a totally new representation of a stop sign no matter where it is in the
15:46
image. I would like to say I'm going to detect this stop sign no matter where it's located.
15:53
And so we'd like to have a layer or a set of features that encodes these principles and right now our fully connected layer totally doesn't. It says that pixel 20 is fundamentally very different from pixel 1 and the pixels in one part of the image are totally different from the pixels in another part of the image and learns weights for those different parts of the images and so we need to come up with some new method that's going to encode these principles
16:16
because we don't have them right now and that's essentially going to be the idea of these filters and max pooling and things like that that go into convolutional neural nets.
16:28
Okay so let's start looking at a concrete example of this.
16:35
In particular, we're going to look at a 1d image. So we're just looking at these two dimensional images. I think we tend to think of our images as being two-dimensional, but here we're going to look at a 1d image. Now first, I want to point out that a one-dimensional signal is not a toy signal. There are a lot of real-life one-dimensional signals. So an example of this is if you look at medical time series, like measuring your heart rate over time, even measuring something like
17:04
the number of COVID cases over time, these are one-dimensional signals. And in fact, people have applied very recently. Here's just one example paper among, I'm sure, many convolutional neural nets to one-dimensional signals. They might not be interpreting them as images so much, but it's still a one-dimensional signal and it's going to look like this, it's just maybe not going to be zeros and ones, it might be something more general. Okay we see this in a lot of other areas
17:31
too. So you can get time series, for instance, in a lot of spatiotemporal applications. So maybe I'm looking at something like monthly temperature at a location. You can get it in genetics, so the genome can be thought of as basically one-dimensional, so you can get data like that. Another way that I might get a signal like this, maybe even a zero-one signal, is maybe I'm doing some astronomy application: I point my telescope at a particular part of the sky and I might be
18:00
interested in understanding is there a light in that part of the sky, maybe it's a star, maybe it's a planet, maybe I'm seeing a satellite passing through and so I could be just basically saying, for every maybe 10 seconds, am I seeing light in that part of the sky and then maybe I want to detect the satellites because they're going to be passing so quickly and so they'll be ephemeral lights, they'll be just going for just a moment as opposed to a longer time. So this is
18:29
just a maybe a motivating example to have in your head for what this 1d image might represent.
18:37
Okay so now what we're going to do is we're going to apply a filter to this image. So what I mean by a filter, here's an example: it's just three numbers and we're going to take a dot product between these three numbers and basically every set of three numbers in the image and get an output, and we're going to call that a convolution.
18:58
Okay so for instance, here are three numbers in my image, here's my filter. So what I'm going to do to get my output is I'm going to multiply the first two: 0 times -1.
19:11
I'm going to multiply the second two numbers: 0 times 1 and add that in. I'm going to multiply the last two numbers: 1 times -1 and I'm going to add that in. So this is just a dot product between the vectors [0, 0, 1] and [-1, 1, -1], the filter. Okay I can compute this dot product, the whole thing is going to be -1, and then I put it in. Now if you have ever heard the words convolution or correlation in the past, you might be thinking “hey, this really sounds like a correlation,
19:41
it doesn't sound like maybe the thing that I've heard of as a convolution in the past” and a genuinely challenging fact of life is that there are just overloading of terms in so many different areas and so you should think of this as being a convolution for the purposes of convolutional neural networks. This is how the term is used here, that doesn't mean that it's always going to be used the same elsewhere. If anything, this is one of those things that makes you really
20:03
realize you have to be careful when talking to somebody else to see does that word mean the same thing that you think it means. I think a great example of this is the word normal: it means like a million different things in a million different areas, across physics, across statistics, across machine learning, everything, and so convolution is just one of those words. But here's what we're
20:22
going to be meaning by convolution: essentially taking this dot product and moving it along. Okay so we're going to do the same thing for the next three set of contiguous pixels here. So we're again going to take this dot product between these two values and we're going to put that in for the output and just to check, could everybody respond to me or anybody who is up for it, respond to me
20:43
in the chat and just say what is the dot product going to be for this particular set of pixels?
20:53
Awesome lots of totally perfect responses, everybody's getting that this is going to be zero.
21:01
Okay and then we just keep doing this: so we we go to the next three set of pixels, we do the next three set of pixels, go to the next three set of pixels and so on and so on until we've gone through every set of three contiguous pixels, every unique set of three contiguous pixels. Okay so now we've performed convolution and a typical thing that we'll do at this point is then, you can think of these as being the pre-activations just like
21:27
we had in our two layer neural network: we had something where we did some kind of combination, some linear combination of weights and our data and we got pre-activations and then we put them through one of these activation functions. So let's say that in this case we'll use a ReLU.
21:46
Okay so if we do that, then we recall that basically that's going to put an output of 0 when our input is 0 or below and it's gonna just put out the output again, put in the same value again, if the input's above 0. So here, we're gonna get a zero, here we're gonna get a zero, we're gonna do zero here, we'll finally get a one and then we'll get a bunch of zeros. Okay and so I think the question that we wanna ask now and this is a real question that again I'm asking to you and
22:13
you should feel very free to respond in the chat is: what does this filter do? We've applied this filter but why like does it actually do anything for us? What does it tell us about this image?
22:31
Great.
22:37
I like the description of it as finding a lonely 1. It's finding an isolated 1. It's finding a 1 that is flanked by two 0s. In this particular case, we're only looking at zero-one data and so basically what's happening is we're doing this convolution and we're gonna get a negative number or a zero if we don't have a one flanked by two zeros and we're going to get something above zero if we do have a one flanked by two zeros. Again, if you think of this in terms of the astronomy
23:06
example, if we get a lot of ones that could be because we're seeing a star and it's just moving slowly because of the rotation of the earth across our telescope, but if we get a single one, maybe that's a satellite zooming across the sky and we want to detect those satellites and so we would be interested in finding where are these isolated ones.
23:28
Okay great, now something that you'll notice here is that if our satellite zoomed by in the first pixel or the last pixel, we wouldn't pick it out right now because what we've done, is we've had to take every three pixels that come together and so we just won't get things at the very edge of this image, at the very ends. Let's start in the beginning and so if we wanted to pick that out, something that we might do is we might add padding to both sides of the image.
23:57
And so a really typical thing to do with padding is to make it zeros. I can't emphasize enough how much everything that we're talking about is a choice and so what you should really do, just like everything we've talked about so far in this class, is you should ask yourself what are you trying to accomplish with any particular application and then ask yourself what type of padding or what choices in general are appropriate for that application. But here,
24:20
if we wanted to detect these satellites, say, then we might say “hey, zero padding is reasonable because we want to be able to detect something that could have been at the edge like a one at the edge.” Okay so in this case, what we'll do is, once we've added our padding, we've added our zeros to the edge, now we have the ability to apply the filter just a little bit further. So now, we can apply the filter here to the three pixels at the beginning,
24:43
including the padded pixel and it's just the same as before: we just do this dot product, now with the padded pixel included and in this case we'll get out of zero.
24:56
Okay and then same thing at the end: we'll just do the dot product now with the padded pixel included because that lets us go a little farther and we get out of zero. And something you'll notice here, this is a very typical thing to try to accomplish with padding, is that usually we're trying to get the image after convolution to be the same size as the image before convolution. So you can say, in some sense, the filter is telling you what's happening in each pixel
25:18
and so now you'll have an answer for each pixel essentially.
25:23
And so once we've done that, we're then going to put these final pixels again through a ReLU,
25:31
and in this case we'll just get some zeros out at the end.
25:38
Okay. Now so far with this filter, we've just applied it as is, but we can apply it with a bias and so what we mean by that, is we take the dot product that we were taking before and we just add the bias term at the end. This is essentially what we've been doing with biases all along: we've been having, for instance in logistic regression, you had essentially a weight vector which was theta and then you added this theta naught at the end. This is exactly like that: you
26:04
have this weight vector which is described by the filter and then we add this bias at the end and so we're doing the same thing here. The only thing that's different is that we're moving this along within our data instead of just applying it to the whole data which is what we would have done with weight vectors before in the class. So here, for instance, we would do this dot product. So the dot
26:26
product hopefully is pretty clearly zero because it's just zero times everything and then we add the bias at the end, that's going to be one, and so we would get one out from this convolution. Okay and then we would do the same thing going forward and we do our dot product, in this case, we get out negative one, we add our bias as one and we get out zero.
26:50
And then we just keep doing this: that fills in all of our numbers and finally we apply the ReLU
26:57
as before. In this case, you see that we get quite a lot more non-zero numbers once we have this bias, so this bias really does change the kind of output that we're getting from our filter.
27:14
Okay. Now in general, we don't have to have a filter that's just made up of integers, we could have absolutely any real weights here. So in general, if we have this filter that has size three, so its length is three, then we would have these three weights. Let's call them w_1, w_2, and w_3, again they can totally be real values,
27:39
the [-1, 1, -1] is nice for illustration, but we could have 0.9, we could have -1.2, you have all these different values for the w's and the same things with the bias b: the b will, in general, be will be something that is real valued as well and we just do the same thing, we do this dot product. Once we have our w_1, w_2, w_3 and our b, we do this dot product with each set of 3 pixels: we add the bias b and we get our convolution and then we can put this through that ReLU layer.
28:12
And it does not have to be a filter of size 3: it could be a filter of size two, it could be a filter of size four, it could be a totally different size filter, it's really a choice and that is part of the architecture, the building of these filters and, in general, of the convolutional neural network.
28:36
Okay. So in this particular case, we did an example of size three and we saw what it did and, in general, it's worth asking what are these filters doing? What are we accomplishing with them and so on?
28:49
Now just to think for a moment, we want to think: what happened here?
28:59
We could have taken this one dimensional image and we could have applied a fully connected neural network layer to create our features, we could have done everything we did last time with the activation, with the activation function and all of that, and so let's ask ourselves: what's fundamentally changed here? In this particular case, supposing we did have a filter of size 3 with a bias b, this is another real question for the chat: how many weights will I have, including
29:28
the bias term? So essentially I'm asking how many parameters do I have when I use this filter?
29:38
Great.
29:44
So in particular, the number of parameters I'm gonna have here, as many of you are pointing out, is four because I have the three weights w_1, w_2, w_3, and I have the bias b.
29:58
So even though I move those around in the image like I'm going to move them to the next pixel and so on, I'm going to use the same weights every time I do that and so essentially at the end of the day. we're going to build up this model that depends on these weights but we're going to try to then learn our parameters. So this is what we've done a million times now: we've built up a model that depends on some parameters and then we try to learn those parameters
30:24
and here if we had this filter, the number of parameters we'd have to learn would be four, because it would be the three weights that we just keep moving along and this bias b.
30:33
Okay, now let's ask ourselves: what if instead we had used a fully connected layer?
30:40
So what would a fully connected layer look like here? First of all, let's notice that I have 10 inputs, those are my pixels, so a data point consists of 10 pixels, in this case x_1, x_2, x_3, up to x_10. In this case, this particular data point is [0, 0, 1, 1, 1, 0, 1, 0, 0, 0], but I would have other data points and things to do like that but the input size is 10. And then my 10 outputs are what I get after this convolution and ReLU and so I could have instead
31:13
had a fully connected layer with 10 inputs and 10 outputs and so now my question for you, which I see some of you are already answering which is awesome, is: how many weights, including biases, do I have for a fully connected layer with 10 inputs and 10 outputs?
31:35
Okay lots of awesome answers here: let's walk through this for anybody who isn’t immediately thinking of this, or even if you are thinking of it just to check your mental map. So if we had this fully connected layer, we would have 10 inputs,
31:53
we'd have 10 outputs and so for every output, we would have to have a weight for every one of the inputs and for our bias. And so that means, for every one of our 10 outputs, we have
32:10
10 weights plus a bias, so 11 total weights. Now we just multiply those together and we get that we'd have 110 weights. And so even for this super simple case, because your images generally aren't going to be 10 pixels long, and this is just going to be way more different if you get to higher levels of pixels, but even for this very simple case, we see that there's a huge difference between the number of weights in the convolutional layer and a fully connected layer
32:38
and the fully connected layer, because everything has to connect to everything else, you get this huge number of weights and biases whereas with this filter we just have a very small number of weights and biases because we're reusing them essentially again and again and again
32:54
and so if this is capable of encoding the things that we want, this could potentially be a really big savings. We don't have to learn 110 parameters. Now we just have to learn four parameters. Another way to think about this, again, is like in terms of that stop sign: if I had to learn this fully connected layer and I wanted to learn the kind of information that was in this filter, I'd have to relearn this filter for every single part of the image,
33:23
I'd have to relearn that there could be a stop sign in every part of the image and so here, instead we're saying “hey I just want to learn this one filter over the whole image” and so that's where the savings is coming from. Now of course, that depends on the thing that we want to learn and encapsulating what we're actually interested in, but if that's true, then this is a really big savings.
33:47
Okay so that's a 1d example, let's go on to something that's a little bit more representative of the types of images that we would see in real life and that's two dimensions,
34:01
at least in terms of images. Again I can't emphasize enough: 1d signals absolutely occur in so many parts of real life, but for images I think we're used to two dimensions.
34:12
Okay so let's look at a 2d image: this is actually our familiar 2d image from before that says “hi”
34:18
and it's just composed of a five by five set of pixels here. And so with our five by five set of pixels, we're now going to apply a filter and now, because we're dealing with 2d images, our filter will also be two dimensional. So in this case, it happens to be a three by three filter,
34:37
and the way that we're gonna apply the filter is very analogous to what we did before in terms of convolution: we're gonna line it up with the part of the image that has the same size, a subset of that image, and then we're going to multiply together the values in each pixel that aligns. So in particular, we're going to start by taking this upper corner: we have a 1 and a -1. When we multiply those together, we get a -1.
35:08
Now, we're going to take the next two sets of pixels that align between the filter and the subset of the image: we have a 0 and a -1. We'll multiply those together and we'll get a 0.
35:18
Now, we'll take the next two sets of pixels that align: it's a 1 and a -1. We'll multiply those together and we'll get a -1. Now we'll go to the next two sets of pixels that align: we have a 1 and a -1. We multiply those together and we get a -1.
35:36
Okay we get a 0 and a 1. We multiply those together, we get a 0.
35:42
1 and -1 is a -1. We keep doing the same thing and so for every one of these nine elements, we're multiplying it together. Now this is really the same operation as a dot product. One way that you could think about this, as doing this dot product, is that you're flattening
36:01
this subset of the image and you're also flattening the filter in the same way to make a vector and then you're doing a dot product between those two vectors. But it's also perfectly legit to think of it as just being you're multiplying the pixels that align in the two things together, you're adding them all up and you're getting a value, in this case, it happens to be negative seven, and so we'll put that down here. And eventually,
36:24
we're going to fill in something that looks like an image by doing these convolution operations.
36:30
Okay and so the next thing we would do is we'd find another unique subset of the image,
36:36
say this one, to perform this convolution. Now what we could do is we could step through again and say “okay, I'm multiplying a 0 and a -1 and I get a 0” and so forth, but something we might notice about this filter is that it's -1s all around the outside and a 1 in the middle, and so one way you could quickly calculate this is to say “oh, I'm going to take negative times the number of ones in my image that aren't in the middle, of my image subset,
37:00
and then I'm going to add one if there's a one in the middle.” So in this particular case, I'll notice there are three ones in my image subset, that thing that's in the blue square, that aren't in the middle and there's one 1 in the middle, so I can take -3 + 1 and my output will be -2.
37:17
Okay so just to check on that, let's say I go to the next step. Can you tell me in the chat what is going to be the convolution of this filter with this subset of the image?
37:30
Awesome looking good. Great, so I think many of you used the shorthand and said “hey, there's four ones that are on the outside, not in the center of this subset of the image, and there's no 1 in the middle and so I'm going to get out a -4.”
37:53
Okay. And then what we would do is we would just keep applying this:
37:59
so we would say “okay, I've made this convolution here, I can go to the next subset of the image that looks like my filter, this next 3 by 3 subset of the image.” I apply this filter convolution, I get out, in this case, -5, -2, -5. I just keep doing these calculations.
38:21
Now something you'll notice here
38:26
is that, at this point, I've applied this filter to my image and now it's much more striking than in the 1d case that I've dramatically reduced the size of my image by going through this convolution step: I went from having a 5x5 image to having a 3x3 image and, again, there are a few things I might be interested in here. One, I might be interested in having my output be the same as my image. In part, though, because I want to say something about the pixels around the outside.
38:55
If I want to be able to talk about these edge pixels, if I want to be able to ask if my filter applies to them, I'm going to need to do something more because right now I can't center my filter at those edge pixels. And so this is what padding will accomplish. If I apply padding all around the outside, then I can center my filter at those edge pixels, and I can pick up something about the edge pixels. So again, it depends what you want to do but this
39:21
would be a very typical thing to do here to be able to say something about those edge pixels and to be able to apply our filter there we would want to apply this padding.
39:32
Okay so once we've applied the padding, you can go in now and there are new sets of these 3 by 3 image segments that I can apply my filter to. So in particular, this is one of them
39:46
and it's just the same operation as before: we say how many ones are around the outside of this little image segment, this is subset of the image: there's one, there's one inside, and so we're going to get zero by adding those together.
40:02
Now we can go to the next step and we can keep doing this and we can fill in, now, an outer layer around what we had before.
40:14
So we just step through, we do the same operations as before and we just have a lot more that we can do because of this padding. And you'll notice, now, once we've applied this padding,
40:28
by design, we now get back something that has the same size as our original image. So our original image was 5 by 5. Now that we can apply this filter with this particular set of padding, we can get this nice 5 by 5 image out. Now it won't always be the case that the right amount of padding is just a padding of size one. That happens to be because of this particular size of filter but it is a very typical thing to use an amount of padding
40:54
that will let you get back something that looks like the size of the original image.
41:01
Okay so now we said “hey, the usual thing is we don't just apply convolution, that's our pre-activation. We then typically will apply some activation function just like we did for our fully connected neural networks that we were talking about before” and so now we're going to apply an activation function again, let's just say it's a ReLU,
41:20
and so if we apply that, we recall that what happens in a ReLU is that anything that is zero or below gets set to zero and anything that's above zero gets set to whatever its value is and so in order to apply a ReLU here, we can just identify what is happening here, what values are above zero, and in this case, there's actually just one, there's actually just a single value that's above zero and so it'll be really easy to know what all of the other values are: they will all be zero.
41:56
Okay so we just set everything to zero. So this is the operation or this is the final activation value, this is what we finally get after doing all of that work,
42:08
both applying the filter and then applying a ReLU to each one of these pixels. Okay and now again, we can ask ourselves “what did this filter do? What was this filter doing? What was the point of this filter? Did it accomplish something that we can explain in words? Again this is a question for the chat, so if you could put your answer in the chat that would be great.
42:42
Good stuff. Okay it found an even more lonely pixel. This one was saying, much like our previous one, much like our previous filter, it's asking—it's not a quarantined one, that's another great way to put it—it found a one that is surrounded by all zeros,
43:06
so we're looking for a one that's surrounded by all zeros. In this particular case, another thing that people have noticed, is that that's the dot on the “i” right because here we have this word “hi” that's spelled out and so essentially what we've done with this filter is we have detected this dot, it was a dot detector. In this case, because we have an image and we're interpreting it as an image, we can really think of that as being a particular dot that we are detecting
43:34
and that's what this filter has detected. Now something that I think is worth is interesting and worth noting here is that we would not have detected the dot on the eye without the padding. So it was specifically the padding that filled out that, hey, there's there's just nothing on this other side of the “i” essentially and so now we can go in and find things and because this was on an edge, it was something that we could really only detect once we had done the padding but once
43:59
we did that, we can detect that dot and another interesting thing about this filter is because it's center aligned, we're seeing that we can really recreate exactly where that dot is because that's exactly the pixel where that that one ended up being after the filter and after the ReLU.
44:18
Okay and again, you don't have to have just negative ones and ones for your filter, you don't have to have integers for your filter and your filter definitely doesn't have to only do dot detection that was just, I think, a nice illustration here.
44:34
So in particular, we'll see in a moment that you can have more general weights. Before we do that, let's just recall that you can have a bias term and that's no different in this case. You can also have a bias term even when you're in the slightly higher dimensional cases, so now we're in this two-dimensional case, and the bias means the same thing in that you essentially do this thing that is like a dot product, you take your subset of your image, you multiply each value
45:05
with the corresponding value in the filter, you add them all up and then finally you add the bias.
45:12
So we have this linear combination between the weights and the filter. Think of the elements in the filters being weights and the data points even with padding and then you add the bias at the end.
45:24
So in this particular case, we would say “okay,
45:28
again there's one, there's a single one that's outside the middle one, so we're gonna get a -1. There's a single one in the center for this blue image, so we're gonna get 1. We add those together, we get 0. We add the bias 2 and so the final output here will be 2.”
45:44
Okay just a quick check, again, let's say that I apply this filter with the bias to this subset of the image, what am I going to get for this output, for the convolution? Just a question for the chat.
46:02
Great, good stuff. Okay so most people here are saying -2. So the first thing that you do, again, is you apply just the convolution without the bias. Once you apply the convolution without the bias, you'll get -4. Then you add the bias of 2 and then you get -2. So now, we have this -2 and finally we're just going to fill in the rest using the same set of operations.
46:40
And without going into that, I'll just point out again, just emphasize again, this filter does not have to be negative ones and ones. In general, we're going to be allowing, and this is actually pretty important that we're going to be allowing, real values for these filter weights. So here you might call them w_1, w_2, w_3 up to w_9, we might call the bias b.
47:01
But these are very much operating like the thetas from before and the bias is like the theta naught or the w's from before and the bias is like the w naught. We've seen this kind of template a bunch of times before, this is just a slightly different way of applying it.
47:20
And it's important that these be real valued because we're going to try to learn them later, we're going to try to do things like gradient descent on them later and so we want them to be able to have a really great range of values and you'll be able to take derivatives and stuff like that and apply them and still have valid values for the w's. Okay, so this is a very general 3 by 3 filter with a bias b. Again, you don't have to have a 3 by 3 filter,
47:47
so we could instead, for instance, have like a 2 by 2 filter,
47:54
so here we would just have four weights w_1, w_2, w_3, w_4. I could have a 4 by 4 filter, it could go from w_1 up to w_16 and certainly that would make more sense as I got to larger images, images that weren't just 5 by 5 images, but that would be something that… Most images that we would encounter would probably be bigger than 5 by 5. Okay question. Oh the question is, sorry, just go for it. Yeah the question was just basically: how do you determine the values of the
48:25
filters such that they allow the CNNs to learn the specific image properties? Great yeah. So I think the premise of this question alone is already a nice point and I want to abstract it. So we are going to determine the weights of the filters, we're going to learn the weights of the filters. So one, the way you can think about a lot of the things that we've been doing in this class is that we first start by specifying a forward model. We say, for instance,
48:51
for linear regression, if I have some data and I have some parameters, I can now decide what do I do for my prediction on that set of data for those parameters. Once I specify that and I specify a loss, I'm able to go back and learn the parameters by trying to basically minimize the training laws
49:09
and we're going to do the same thing here. We're going to say, first, we're going to build up this forward model that says “hey, if I had some data and I had some parameters, what is my label on a new data point going to be?” And so that's essentially what we're working on right now, that's what we're going through right now, and then once I have done that and I have specified a loss—and at this point you're familiar with a lot of losses that we might choose,
49:30
like if I'm doing regression, I might do the squared error loss. If I'm doing classification, I might do something like a negative log likelihood loss or something along those lines—and then once I have that, once I have the loss on my training data, I can minimize it. And so this will be no exception to that, it's always the case that the math for minimization might look a little bit different but even then we tend to be applying things like gradient descent,
49:54
stochastic gradient descent for that minimization and so a lot of it looks very similar, it's just like what do those gradients look like? And that is exactly what we'll be doing here too.
50:10
Okay so the reality is we think of all the images we see as being two-dimensional but the reality is most images that we see are probably best described as three-dimensional and the reason for that is that they don't just have a width and a height, which is what we're very used to thinking about, but they also have a depth because, if they are in color, they have the encoding of that color. So for instance, we might have a red, a green, and
50:34
a blue value for each one of our pixels and so as soon as we have those three values for each pixel, we further have the depth. And in particular, if it's red green blue, that depth has size three
50:47
and so even though when you hear 3d image, I think you tend to think like oh something that looks 3d or has some cool property, what I really mean here is an image that you're familiar with, like it's like the computer screen that you have right now is a 3d image. What you're looking at right now is a 3d image just because each pixel has these red green blue values that describe the color or
51:09
something similar to that. And so for that reason, we often want to think about 3d images and you can think of these 3d images as being a generalization of concepts that you've seen before. So when I think of a structure that is a list of numbers, I'm thinking of a 1d image, it's a vector. When I think of a structure that is a list of list of numbers and they all have the same size,
51:41
then I'm typically thinking of a 2d structure and it's a matrix. So I could have something like, for a vector, I would just have maybe the width of that vector or the length of that vector, so it has one dimension, and I think of a matrix, I think of the width and the height of the matrix and that describes the matrix the size and that's two dimensions. And now a tensor is just the three-dimensional generalization: I have a width, I have a height, but I also have a depth.
52:05
Now that's a little bit hard for me to draw all the numbers for and part of that is because, again, we as humans, as we said before, can only see in two dimensions and so even getting into three dimensions sometimes can be a little bit challenging but this is certainly an idea that comes up a lot in image processing and beyond. For images, one reason that it comes up a lot is what we've just seen that typically images are expressed in a way that they also have a depth,
52:31
specifically just because of that color encoding. We'll see in a moment why we also might care about these tensors even if we were working just with two-dimensional images it's gonna come up.
52:42
Because tensors are so big and image processing and they come up in neural nets and convolutional neural nets and so on you see this word bandied about a lot. So for instance an example of this would be the software or the platform Tensorflow. So this is very common to use for something like neural nets but also, more generally, machine learning and you can see where that logo might be coming from and why it's being called Tensorflow. And just as we had one-dimensional filters and
53:13
two-dimensional filters, we could also have three-dimensional filters. And so I'm just gonna give a geometric intuition for this rather than running through the numbers like we did before, but it's all the same. We would take our filter, we would go to our image, and we would say “hey, for this image, where can we put a filter of this size to align with the image?”
53:38
And then we would step through the pixels of that image or the items that are in the width and the height and the depth, the elements of the tensor. And so we would just step slowly through just like we did before, we would apply again this notion of a convolution and we would get out again something that has this width and height and depth, especially if we apply padding.
54:01
Okay we would do this all throughout even down at the bottom and we would get out our output. Okay so now I said that there's another way that tensors arise and so let me mention that as well.
54:14
So even if I have a two-dimensional image, something that I might do is I might apply multiple filters. So maybe I apply my first filter and it's something like a point detector like we had before and maybe let's call that filter F_1. And maybe I apply another filter, that's like an edge detector, we could call that F_2. Then maybe I apply another filter which is maybe a different type of edge detector, maybe one of them is a horizontal edge detector
54:42
and one's a vertical edge detector, and so I have all these different filters. And you'll notice now that, when I put these filters together, I have a bunch of two-dimensional images. So when I put them together, I have a depth which makes this into, in some sense, a certainly a three-dimensional structure and you might think of it as a three-dimensional image.
55:02
And so this collection of filters in the layer, all these different filters F_1 and F_2 and F_3 might be called a filter bank and each of their outputs might be called a channel.
55:16
It's just some terminology but we see that this is another way that tensors arise. So even if you're working just with one-dimensional images or two-dimensional images, you might well find yourself dealing with tensors and doing further operations on these tensors after this particular layer.
55:39
Okay so at this point, we have these filters, they detect various things in the image potentially and now we want to ask what do we do with that. Maybe I've defined my filter that detects dots or detects edges or whatever it is and it's done its work and it's detected some dots or some edges and now I want to ask, okay I've detected these things, what do I do with them? Well one thing
56:06
that I might do is I might pull them together, I might say “hey, in this little area of the image, did I detect any dots? Did I detect any edges?” And I asked myself over this whole area: what did I detect? And so this will be the idea of a max pooling layer. So here's a two-dimensional example: you can think of this as being the output from our previous layer. So remember, one thing that we're doing in this lecture is we're building up lots of layers,
56:32
we're not keeping ourselves to just one layer and so we're thinking of, “hey, I've applied my convolutional layer with its filter in this case, I applied my ReLU activation function, and this is what I might have gotten out and I notice there are actually a few ones that popped up, something was detected.” It would be nice to say, “hey in this little area of the image, was anything detected?” And so that's what we're going to be doing with a max pooling layer.
57:01
So in particular, with the max pooling layer, it's going to return the max of its arguments. So we're going to do a similar thing to what we did before: we're going to specify a subset of the image size, let's say size 3 by 3 and this will often be a square and so we might just call it size 3 for shorthand.
57:23
And so what we'll do, is we'll again go to our image, we'll take a little subset of the image. So here's that subset that we had before
57:33
and we'll ask ourselves, now, to apply the max pooling layer. So in the max pooling layer, we just look at the maps of the arguments and so in this case, when we apply our max pooling instead of our filter, we're just going to get a zero because the max of all these zeros is zero.
57:51
Now we could do the same thing that we did before: we could step over by one and apply max pooling again. And again, it's just a bunch of zeros, so we're gonna get an output of a zero. We could step over one, apply max pooling again and say what's the max of all these numbers? Now we've got a one, so we're gonna output a one. We could step over again and say what's the max of these numbers? It's 1. And we could do the same thing: we could step over everything, even at the very
58:15
bottom here, and just ask what's the max in this little subset of the image? And certainly that was something that we did with the filter was we just stepped through the image and asked “hey, if I apply this filter, what happens? And now we're saying, “hey if I apply this max pooling, what happens? But if you think about it, kind of what we're doing with the max pooling is we're asking
58:35
ourselves did I detect something by taking the max and asking if it's non-zero. I'm asking was there any detection and so from that perspective, when I just step by one, I'm kind of being redundant, like I'm asking the same question of a lot of pixels. I might instead want to just say “hey, in this little area of the image, did I detect anything? Now in this little area of the image that's separate, did I detect anything? Now in this area of the image that's separate,
59:01
did I detect anything?” And so I might do that instead. So if I do that, if I skip ahead, then I'll say that I have a different stride. So you can think of all of the things that we did so far, the max pooling on this slide but also the filters on the previous slides, is having a stride of one. And what I mean by stride of one is that we looked at a subset of the image that fit our filter or our max pooling and then we stepped over one pixel and then we did it again.
59:31
If we had a different stride, we would step over a different number of pixels.
59:36
So let's look at an example of that. Instead of a stride of one, let's instead look at a stride of three. So in this case, a stride of three would be really natural because our max pooling has size three and so we're going to ask of one set of the image whether something happened and then we'll go to a different part of the image. So let's look at a stride of three: what would the output be like here? How would things be different if we had a stride of three?
1:00:04
Well we'll still start in the corner of the image and we'll say, “okay, this is our max pooling size. It's 3 by 3. I'm gonna apply max pooling and I'm going to get a zero because the max of everything that's in that 3 by 3 area is zero.” And now,I'm gonna start stepping.
1:00:24
I'm gonna step by one, I'm gonna step by two, I'm gonna step by three,
1:00:31
and now that I've made three steps, three pixels over because this is a stride of three, I will again apply max pooling and I will say “okay, for this region of the image did I detect anything? Is there anything that's above zero?” In fact, there is: the max of this set of things is one.
1:00:49
And I can do the same thing in the other direction. I can say “okay, this is where I was applying max pooling before and now I'm going to step by one, I'm going to step by two, I'm going to step by three.” And now I will apply max pooling again and I will say “okay, what's the max of what we have in the pixels that we're seeing here? We have a bunch of zeros, but we have a one and so the max will be one.”
1:01:12
And then finally I step by one, I step by two, I step by three. I could have also stepped from above, it'll be the same thing, and now that I've taken my stride of three, I can again ask okay well what happens when I do this max pooling? I get the max of all the zeros and that's going to be zero.
1:01:33
Okay now in this case, we've dramatically reduced the size of the image, but with a max pooling layer, that's actually something that we're often looking for. So with the filtering we're saying “hey, were there things that happened at any of the pixels that we have?” With max pooling we're really combining this information. We're saying “hey, in these different areas, did something happen?” And maybe there aren't so many
1:01:56
areas as before and so we get something that's quite a bit smaller potentially. And you can imagine doing this to build up larger and larger features essentially.
1:02:12
You can also use stride with filters. We didn't talk about that in this lecture but it's certainly something you can do: you could apply a filter after some strides. But I think that the way that we've talked about it is a natural way, a way that you might choose to use strides and filters and max pooling. Something that's also worth noting is that there are just no weights in max pooling, there was nothing here that was a value that we were interested in changing,
1:02:40
that we were trying to learn, we set these sizes, we set these strides, there's no real valued parameter that's sitting around here. And so in this case, we're not going to have something that we're going to go back later and say “okay, let's gradient descent our way to learn some parameters, there just weren't any here.”
1:03:05
Okay so let's bring this all together then: what would a typical architecture look like for convolutional neural nets? Well, first you have your input. So here you can see there's this picture of a car, that could be potentially an input to a convolutional neural network. And here, you can see that there is a filter—in this case, in three dimensions—because you can see that car has the three dimensions,
1:03:30
probably because it's some kind of color picture so it has a depth of three for each of its pixels.
1:03:38
And you can see that filter going over it and coming out with some output in a convolutional and ReLU layer. Now once we have that convolutional and ReLU layer, a typical thing is to have a max pooling layer and you can see again that that max pooling layer is going to dramatically reduce the size of the layer. That would be something that we might see. Now once we've done that, maybe we've detected edges or we've detected dots or we've done whatever now,
1:04:05
we can have another convolutional and ReLU layer and that might detect some combination of edges, there's some combination of dots, because now each of our pixels is essentially a detection of an edge or a dot so we can get these slightly higher order features. Now we have a max pooling layer and now we can detect combinations of combinations of edges or combinations of combinations of dots
1:04:26
and our hope is that, over time, these become features that really matter, maybe we're starting to detect a tire or we're detecting something that is actually a useful feature for what we're doing. And so we do this a bunch, we do these layers a bunch. That's the dot, dot, dot after the max pool. Finally, eventually, we have a bunch of values and we're going to treat those as our final features and so we're going to take them out of this tensor, we're going to
1:04:53
flatten them into a vector, that is to say we'll just take the values of the tensor in some order,
1:05:00
treat those as features in a typical classification and so now this finer layer is just exactly what we've been doing for many classes. We have a fully connected layer,
1:05:12
so that could be interpreted as logistic regression, it could be interpreted if we were doing just regular regression as a linear regression. In this particular case, this is a classification problem and so we might be doing some form of logistic regression, in this case with multiple classes. So we have this fully connected layer
1:05:30
and then finally we apply, in this case, a softmax function to get our prediction. So because we're using softmax, our predictions will be soft predictions, they'll be saying: what's the probability of this being a car? What's the probability of this being a truck? What's the probability of this being a van or a bicycle or whatever the set of possible options are?
1:05:52
Okay and so we now just have a lot more that's going on in the feature learning. In the previous class, we had just one layer that was doing the feature learning, it was this fully connected layer and it was a fully connected neural network there, and we saw what to do with that. And here, there's just a much richer set of layers that are going into that feature learning and when we say that there's learning happening that's
1:06:17
really when we choose those weights later. So first we build up, again, this model and then we're going to choose the weights and then what hasn't changed is that that output layer is still the same, it's still doing classification, and still doing the thing that we saw in logistic regression and still doing the thing that we saw in our neural networks for the output layer in our previous neural networks lecture and so that's all the same, it's really this feature construction
1:06:42
that has changed. And so we have specified at this point a way to take a data point x,
1:06:50
so that's our image. In this case, it's the image of a ca in this particular slide, but in general it's whatever image input we have and we specified a way to turn it into a prediction and we might call that neural net, this is a neural net, but of course we've seen that there were a lot of choices that went in along the way and so to fully specify this, you'd have to say: what was the architecture
1:07:13
you used? What was the filter set that you used? What was the filter bank that you used? Did you use max pooling? What were all of your layers and how big were they? And so there's a lot more to specify than we had to specify in the case of linear regression or logistic regression. But once we do that, we have a prediction. It depends on the data point that we put in and it depends on the parameters. And so now we can go back and learn those parameters.
1:07:42
And so let's just briefly recap how this relates to the things that we've seen in this course. There's really this familiar pattern, at this point, hopefully familiar.
1:07:52
So in particular, we start by choosing how to predict a label. It depends, so this is the label for a particular data point, so it depends on the features of that data point
1:08:01
and it depends on the parameters in general. So an example of this, let's say we had our data point, an example of this was logistic regression. We said for the ith data point, we'll use logistic regression and that'll tell us the prediction for the ith point and that might take the form of probabilities over particular classes. Now once we take this over all of the different points, one through little n, and once we choose a loss between our prediction, our guest label
1:08:32
and our actual label, we can get a training loss. So we can get an objective to minimize.
1:08:41
And then the final thing we've been doing is we've been coming up with ways to choose the parameters to try to minimize the training laws. Now we might also have a regularizer in here, but we're trying to choose a set of parameters that minimize maybe some variant of a training loss.
1:09:00
Okay and then we did this a bunch more times in this class. We said, “okay, another option is linear regression.” We chose how to predict a label, we said we're going to use linear regression. That tells us, given a set of data features and a set of parameters, what is our prediction for the ith point? We chose a loss for instance, it might be squared error loss in the case of linear regression. Once we have that loss, we can go over every single point,
1:09:25
not just one particular point but one up to little n and we can calculate our training loss for a particular set of parameters. And then we can move around in the parameter space in step three to try to choose parameters that are better. And our notion of better is minimizing this thing that's like a training loss. And I say like just because there could also be a regularizer there. Okay and then we did it again. So all we're doing in neural nets and convolutional neural
1:09:54
nets is setting up a different model. We're saying, “hey, now the way that I'm going to predict for the ith data point is I'm going to use a particular type of neural net.”
1:10:06
And that's going to depend on some parameters and it's going to depend on the data point.
1:10:11
Once I have chosen a loss, and these will be familiar losses, again something like squared error loss for regression or negative log likelihood loss for classification, then I will have a training loss over all my points, that'll be a function of my parameters. And so I can ask myself: if I move around in this parameter space, can I do better? Can I get a lower training loss? And that's what we're doing with things like gradient descent and
1:10:38
stochastic gradient design: we're just moving around and asking, “hey, can we get a better loss by doing that?” Now the one thing that is a bit different in neural nets versus logistic regression or linear regression is that you can't just say neural nets and you fully specify what you're doing. As we've seen at this point, there's a lot of different things that you can do within neural nets: you could do a two layer fully connected way of specifying the features,
1:11:06
you could instead learn the features with this convolutional neural net. There are actually lots of other things that you could do that we haven't even talked about and so one difference is to fully specify what you're doing, to fully specify your model, to be really clear and reproducible, you want to say all of those details of your architecture and the best thing you can do, of course, in general to be reproducible, is to provide code.
1:11:33
Okay so this is a familiar pattern and now let's just look in a tiny bit more depth at what might have changed when we're applying something like gradient descent or stochastic gradient descent in the case of convolutional neural nets versus the neural nets that we've previously seen.
1:11:50
So in order to do that, we're going to look at a particularly simple case. Just as an example,
1:11:56
we're going to look at a regression case with a single filter that has size 3. We'll ignore max pooling for the moment. We'll assume that we have padding so that our output image from this first layer of constructing a feature is the same size as our original image and we'll assume our data points have dimension five, they have five features, again just for concreteness. Okay so here's what a forward pass aka just specifying the model might look like.
1:12:32
So first we're going to apply our filter. So this is what we saw before in the images and now we're just writing out the words for what that might look like. So we're going to say that the ith preactivation is the application of the filter which is given by W^1
1:12:49
and then we dot product that with the X elements but they are changing. So the ith pre-activation has gotten from the elements of X that are around i essentially. And because this has a filter of size 3, that's why we're looking at i - 1, i, and i + 1 because those are the three pixels.
1:13:09
And this is a one-dimensional image because it's just dimension 5 by 1.
1:13:14
Okay we apply our activation function, our ReLU, to get out our outputs. These are basically our learned features once we learn the W's. Now, we're going to have our layer that does either the regression or the classification. Because we're doing regression, we don't have to worry about activation function because it's just going to be the identity, so that's implicit in here, and we're just going to have this linear combination with the A_i’s. In this particular case, there are
1:13:41
no biases, I'm just going to ignore them for the moment and then finally we have some loss. Because we're doing regression, we might choose a squared error loss, but of course here, we're getting into this notation we've been seeing this class where A^2, that's just a superscript, but the 2 that's a superscript outside is a square and so you just want to keep those separate in your mind.
1:14:04
Okay so let's briefly check, first of all, what is going to be the size of Z^1 as a vector? So once we apply this filter, we get our pre-activations. Z^1 is a vector, what is its size? This is a question for the chat.
1:14:38
Okay great so a number of people are saying it's 5 by 1. It's gonna be, again, the size of our image, specifically because we chose enough padding to recreate the size of our image. So if we just applied a filter of size three to our image and we didn't do any padding, this wouldn't necessarily be the case, but because we chose enough padding, we will get this 5 by 1, so by design. So now we apply the ReLU, so we're gonna get a 5 by 1 vector there as well. Finally we apply
1:15:08
this weeding and so we get out now a one by one vector because this is just the label for our data point in regression, we're just giving it some label, we're assuming W^2 is going to be basically 5 by 1, so this will just be a dot product and we'll get out of label. And then finally, our loss is also a scalar. Now something you might notice here is that three doesn't seem to appear anywhere and that's because it's in the filter. So it's in the
1:15:36
definition of Z_i and each i, we're essentially taking a dot product over a vector of size three.
1:15:45
Okay so now if you were doing stochastic gradient descent,
1:15:50
in order to try to figure out how to minimize this loss over all the data points, not just a single data point, but over all the data points, then you might do stochastic gradient descent to try to minimize this loss and you might do it for both W^1 and W^2. So I'm saying part of the derivative for SGD, because I'm going to focus for the moment just on W^1. Of course you would do both if you're doing regular sgd, but I'm just going to focus on the part with respect to W^1.
1:16:20
Now just as before, we can apply, so just as we did for the case of certain vanilla neural networks, fully connected neural networks, we can apply the chain rule.
1:16:32
And so in particular, we're going to say “okay, the loss is a function of A. It's a function of both A^2 and A^1, but let's go straight to A^1.” You could even apply the chain rule within this term but I'm just going to say it's a function of A^1, so we can take the derivative of the loss with respect to A^1. A^1 is a function of Z^1, so I can take the derivative of A^1 with respect to Z^1 and Z^1 is a function of W^1, so I can take the derivative of Z^1 with respect to W^1.
1:16:57
Okay and now we want to check that our dimensions work out. So what is the dimension of the gradient of the loss with respect to W^1, can anybody say in the chat?
1:17:23
Okay so this one's a little bit tricky.
1:17:26
Remember W^1, we're taking a dot product of that with a three long snippet of x. And so W^1 has got to be size three. Now a way that you can think about these derivatives, in general, is that you're taking the derivative, you're taking this gradient of a vector with respect to a vector.
1:17:49
The vector on top is going to tell you the second size and the vector on the bottom is going to tell you the first size. So because W^1 has length three, we're going to have three by something. Because the loss is a scalar, so it has size one, this is gonna be 3 by 1
1:18:07
and you can play this game for the rest of these. So Z and W, well W^1 is on the bottom, it has length three, so it's going to be a three by something vector. Z is on the top, it has length five, so it's going to be a 3 by 5 vector.
1:18:25
This is going to be the length of Z by the length of A, so it'll be a 5 by 5 vector. And finally the loss dA is going to be the length of A by the length of the loss, so it's going to be a 5 by 1 vector and then you can just check out all those inner dimensions agree and this is something that will end up with a 3 by 1 vector.
1:18:47
Okay so let's look at what would dZ/dW look like here? Well we already said it's going to be 3 by 5, so it's going to be the three elements of W by the five elements of Z.
1:19:01
If we want to fill in one of its elements, that's going to be the particular element of Z, d particular element of W. So in this case, essentially, the second element of Z with respect to the third element of W.
1:19:19
Now in order to calculate that, it will help us to remember what is Z as a function of W. And so here, I've just written out that forward pass, Z_2, as a function of the W's and the X's with the i’s filled in. So I'm taking i = 2 here from what's above and so if you look at this formula, can you tell me what's the derivative of Z_2 with respect to W_3?
1:19:54
Okay great, lots of great answers here. It's going to be X_3 because we just read that off straight from the formula. You can do the same thing and it's worth checking for yourself that you're going to get X_2 and X_1 here and you can fill in all the other elements in this matrix and something that's pretty interesting here is like you're seeing a wave of X's, you're getting a lot of x's repeated in a way that we didn't see before for neural networks,
1:20:18
and that's because we're moving this filter along in a way that we weren't before.
1:20:23
Okay, I told you that X has dimension five and so what does X_0 or X_6 mean if X just goes from 1 to 5? Can you tell me very quickly in the chat?
1:20:38
Great. It's the padding. So what we're doing when we're adding padding is we're essentially increasing the size of X just for the purposes of our algorithm and that's what's happening here and that's what I mean by X_0 and X_6, I mean that we're bringing in the padding and so if we didn't have that padding, then we would get a different size derivative here and we wouldn't get quite the same answer.
1:21:02
Okay so this is just to illustrate how backpropagation is mostly the same. Certainly the way that you apply it is the same: you're just calculating these derivatives, you're just doing gradient descent as before for neural networks, but the results of that are going to look a little bit different and in particular this dZ/dW term is going to look a little bit different and it's worth reflecting on that. Okay so at this point we
1:21:28
talked about not just neural networks of one layer like logistic regression or linear regression, not just neural networks of two layers with a fully connected network which is what we did last time, but deep neural networks. We have covered now deep neural networks, there are multiple hidden layers that construct the features. We've done this with a real focus on images and certainly convolutional nets are really widely used in image processing and so we have built up to this
1:21:57
thing that is really responsible, that is what is behind all this excitement in AI, and so hopefully you found that to be an exciting thing. Okay I'll catch you next time, enjoy the rest of your week.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat