Skip to main content
Loading Explore guides…
Neural Net Training · Acedra
Home
Explore
Computer Science
Neural Net Training
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
228 segments
Copy
Hide video
0:11
- Okay, let's get started.
0:16
Okay, so today we're going to get into some of the details about how we train neural networks.
0:23
So, some administrative details first. Assignment 1 is due today, Thursday, so 11:59 p.m. tonight on Canvas.
0:33
We're also going to be releasing Assignment 2 today, and then your project proposals are due Tuesday, April 25th. So you should be really starting to think about your projects now if you haven't already. How many people have decided what they want to do for their project so far? Okay, so some, some people, so yeah, everyone else, you can go to TA office hours if you want suggestions and bounce ideas off of TAs.
1:06
We also have a list of projects that other people have proposed. Some people usually affiliated with Stanford, so on Piazza, so you can take a look at those for additional ideas.
1:20
And we also have some notes on backprop for a linear layer and a vector and tensor derivatives that Justin's written up, so that should help with understanding how exactly backprop works and for vectors and matrices. So these are linked to lecture four on the syllabus and you can go and take a look at those.
1:45
Okay, so where we are now. We've talked about how to express a function in terms of a computational graph, that we can represent any function in terms of a computational graph. And we've talked more explicitly about neural networks, which is a type of graph where we have these linear layers that we stack on top of each other with nonlinearities in between.
2:10
And we've also talked last lecture about convolutional neural networks, which are a particular type of network that uses convolutional layers to preserve the spatial structure throughout all the the hierarchy of the network. And so we saw exactly how a convolution layer looked, where each activation map in the convolutional layer output is produced by sliding a filter of weights over all of the spatial locations in the input. And we also saw that usually we can have many filters per layer,
2:42
each of which produces a separate activation map. And so what we can get is from an input right, with a certain depth, we'll get an activation map output, which has some spatial dimension that's preserved, as well as the depth is the total number of filters that we have in that layer. And so what we want to do is we want to learn the values of all of these weights or parameters, and we saw that we can learn our network parameters through optimization, which we talked about little bit earlier
3:11
in the course, right? And so we want to get to a point in the loss landscape that produces a low loss, and we can do this by taking steps in the direction of the negative gradient. And so the whole process we actually call a Mini-batch Stochastic Gradient Descent where the steps are that we continuously, we sample a batch of data. We forward prop it through our computational graph or our neural network. We get the loss at the end. We backprop through our network to calculate the gradients.
3:42
And then we update the parameters or the weights in our network using this gradient.
3:50
Okay, so now for the next couple of lectures we're going to talk about some of the details involved in training neural networks. And so this involves things like how do we set up our neural network at the beginning, which activation functions that we choose, how do we preprocess the data, weight initialization, regularization, gradient checking. We'll also talk about training dynamics. So, how do we babysit the learning process? How do we choose how we do parameter updates,
4:19
specific perimeter update rules, and how do we do hyperparameter optimization to choose the best hyperparameters? And then we'll also talk about evaluation and model ensembles.
4:33
So today in the first part, I will talk about activation functions, data preprocessing, weight initialization, batch normalization, babysitting the learning process, and hyperparameter optimization.
4:47
Okay, so first activation functions.
4:52
So, we saw earlier how out of any particular layer, we have the data coming in. We multiply by our weight in you know, fully connected or a convolutional layer. And then we'll pass this through an activation function or nonlinearity. And we saw some examples of this. We used sigmoid previously in some of our examples. We also saw the ReLU nonlinearity. And so today we'll talk more about different choices for these different nonlinearities and trade-offs between them.
5:22
So first, the sigmoid, which we've seen before, and probably the one we're most comfortable with, right? So the sigmoid function is as we have up here, one over one plus e to the negative x. And what this does is it takes each number that's input into the sigmoid nonlinearity, so each element, and the elementwise squashes these into this range [0,1] right, using this function here. And so, if you get very high values as input, then output is going to be something near one.
5:51
If you get very low values, or, I'm sorry, very negative values, it's going to be near zero. And then we have this regime near zero that it's in a linear regime. It looks a bit like a linear function. And so this is been historically popular, because sigmoids, in a sense, you can interpret them as a kind of a saturating firing rate of a neuron, right? So if it's something between zero and one, you could think of it as a firing rate. And we'll talk later about other nonlinearities,
6:18
like ReLUs that, in practice, actually turned out to be more biologically plausible, but this does have a kind of interpretation that you could make.
6:30
So if we look at this nonlinearity more carefully, there's several problems that there actually are with this. So the first is that saturated neurons can kill off the gradient. And so what exactly does this mean? So if we look at a sigmoid gate right, a node in our computational graph, and we have our data X as input into it, and then we have the output of the sigmoid gate coming out of it, what does the gradient flow look like as we're coming back? We have dL over d sigma right?
7:02
The upstream gradient coming down, and then we're going to multiply this by dSigma over dX. This will be the gradient of a local sigmoid function. And we're going to chain these together for our downstream gradient that we pass back. So who can tell me what happens when X is equal to -10? It's very negative. What does is gradient look like? Zero, yeah, so that's right. So the gradient become zero and that's because in this negative, very negative region of the sigmoid,
7:35
it's essentially flat, so the gradient is zero, and we chain any upstream gradient coming down. We multiply by basically something near zero, and we're going to get a very small gradient that's flowing back downwards, right? So, in a sense, after the chain rule, this kills the gradient flow and you're going to have a zero gradient passed down to downstream nodes.
7:59
And so what happens when X is equal to zero?
8:05
So there it's, yeah, it's fine in this regime. So, in this regime near zero, you're going to get a reasonable gradient here, and then it'll be fine for backprop. And then what about X equals 10?
8:19
Zero right. So again, so when X is equal to a very negative or X is equal to large positive numbers, then these are all regions where the sigmoid function is flat, and it's going to kill off the gradient and you're not going to get a gradient flow coming back.
8:37
Okay, so a second problem is that the sigmoid outputs are not zero centered. And so let's take a look at why this is a problem. So, consider what happens when the input to a neuron is always positive. So in this case, all of our Xs we're going to say is positive. It's going to be multiplied by some weight, W, and then we're going to run it through our activation function. So what can we say about the gradients on W?
9:12
So think about what the local gradient is going to be, right, for this linear layer. We have DL over whatever the activation function, the loss coming down, and then we have our local gradient, which is going to be basically X, right? And so what does this mean, if all of X is positive?
9:36
Okay, so I heard it's always going to be positive. So that's almost right. It's always going to be either positive, or all positive or all negative, right? So, our upstream gradient coming down is DL over our loss.
9:50
L is going to be DL over DF. and this is going to be either positive or negative. It's some arbitrary gradient coming down. And then our local gradient that we multiply this by is, if we're going to find the gradients on W, is going to be DF over DW, which is going to be X.
10:08
And if X is always positive then the gradients on W, which is multiplying these two together, are going to always be
10:18
the sign of the upstream gradient coming down. And so what this means is that all the gradients of W, since they're always either positive or negative, they're always going to move in the same direction. You're either going to increase all of the, when you do a parameter update, you're going to either increase all of the values of W by a positive amount, or differing positive amounts, or you will decrease them all. And so the problem with this is that,
10:46
this gives very inefficient gradient updates. So, if you look at on the right here, we have an example of a case where, let's say W is two-dimensional, so we have our two axes for W, and if we say that we can only have all positive or all negative updates, then we have these two quadrants, and, are the two places where the axis are either all positive or negative, and these are the only directions in which we're allowed to make a gradient update. And so in the case where,
11:19
let's say our hypothetical optimal W is actually this blue vector here, right, and we're starting off at you know some point, or at the top of the the the beginning of the red arrows, we can't just directly take a gradient update in this direction, because this is not in one of those two allowed gradient directions. And so what we're going to have to do, is we'll have to take a sequence of gradient updates. For example, in these red arrow directions that are each in allowed directions,
11:49
in order to finally get to this optimal W.
11:53
And so this is why also, in general, we want a zero mean data. So, we want our input X to be zero meaned, so that we actually have positive and negative values and we don't get into this problem of the gradient updates. They'll be all moving in the same direction. So is this clear? Any questions on this point?
12:17
Okay.
12:22
Okay, so we've talked about these two main problems of the sigmoid. The saturated neurons can kill the gradients if we're too positive or too negative of an input. They're also not zero-centered and so we get these, this inefficient kind of gradient update. And then a third problem, we have an exponential function in here, so this is a little bit computationally expensive. In the grand scheme of your network, this is usually not the main problem, because we have all these convolutions
12:49
and dot products that are a lot more expensive, but this is just a minor point also to observe.
12:59
So now we can look at a second activation function here at tanh. And so this looks very similar to the sigmoid, but the difference is that now it's squashing to the range [-1, 1]. So here, the main difference is that it's now zero-centered, so we've gotten rid of the second problem that we had. It still kills the gradients, however, when it's saturated. So, you still have these regimes where the gradient is essentially flat and you're going to kill the gradient flow.
13:29
So this is a bit better than the sigmoid, but it still has some problems.
13:37
Okay, so now let's look at the ReLU activation function. And this is one that we saw in our examples last lecture when we were talking about the convolutional neural network. And we saw that we interspersed ReLU nonlinearities between many of the convolutional layers. And so, this function is f of x equals max of zero and x. So it takes an elementwise operation on your input and basically if your input is negative, it's going to put it to zero. And then if it's positive,
14:09
it's going to be just passed through. It's the identity. And so this is one that's pretty commonly used, and if we look at this one and look at and think about the problems that we saw earlier with the sigmoid and the tanh, we can see that it doesn't saturate in the positive region. So there's whole half of our input space where it's not going to saturate, so this is a big advantage. So this is also computationally very efficient. We saw earlier that the sigmoid has this E exponential in it.
14:43
And so the ReLU is just this simple max and there's, it's extremely fast. And in practice, using this ReLU, it converges much faster than the sigmoid and the tanh, so about six times faster. And it's also turned out to be more biologically plausible than the sigmoid. So if you look at a neuron and you look at what the inputs look like, and you look at what the outputs look like, and you try to measure this in neuroscience experiments, you'll see that this one is actually a closer approximation
15:16
to what's happening than sigmoids. And so ReLUs were starting to be used a lot around 2012 when we had AlexNet, the first major convolutional neural network that was able to do well on ImageNet and large-scale data. They used the ReLU in their experiments.
15:37
So a problem however, with the ReLU, is that it's still, it's not not zero-centered anymore. So we saw that the sigmoid was not zero-centered. Tanh fixed this and now ReLU has this problem again. And so that's one of the issues of the ReLU. And then we also have this further annoyance of, again we saw that in the positive half of the inputs, we don't have saturation, but this is not the case of the negative half. Right, so just thinking about this a little bit more precisely.
16:07
So what's happening here when X equals negative 10? So zero gradient, that's right. What happens when X is equal to positive 10? It's good, right. So, we're in the linear regime. And then what happens when X is equal to zero?
16:26
Yes, it undefined here, but in practice, we'll say, you know, zero, right. And so basically, it's killing the gradient in half of the regime.
16:38
And so we can get this phenomenon of basically dead ReLUs, when we're in this bad part of the regime. And so there's, you can look at this in, as coming from several potential reasons. And so if we look at our data cloud here, this is all of our training data,
16:59
then if we look at where the ReLUs can fall, so the ReLUs can be, each of these is basically
17:09
the half of the plane where it's going to activate. And so each of these is the plane that defines each of these ReLUs, and we can see that you can have these dead ReLUs that are basically off of the data cloud. And in this case, it will never activate and never update, as compared to an active ReLU where some of the data is going to be positive and passed through and some won't be. And so there's several reasons for this. The first is that it can happen when you have bad initialization.
17:37
So if you have weights that happen to be unlucky and they happen to be off the data cloud, so they happen to specify this bad ReLU over here. Then they're never going to get a data input that causes it to activate, and so they're never going to get
17:55
good gradient flow coming back. And so it'll just never update and never activate. What's the more common case is when your learning rate is too high. And so this case you started off with an okay ReLU, but because you're making these huge updates, the weights jump around and then your ReLU unit in a sense, gets knocked off of the data manifold. And so this happens through training. So it was fine at the beginning and then at some point, it became bad and it died. And so if in practice,
18:24
if you freeze a network that you've trained and you pass the data through, you can see it actually is much as 10 to 20% of the network is these dead ReLUs. And so you know that's a problem, but also most networks do have this type of problem when you use ReLUs. Some of them will be dead, and in practice, people look into this, and it's a research problem, but it's still doing okay for training networks. Yeah, is there a question? [student speaking off mic]
19:02
Right. So the question is, yeah, so the data cloud is just your training data. [student speaking off mic]
19:18
Okay, so the question is when, how do you tell when the ReLU is going to be dead or not, with respect to the data cloud? And so if you look at, this is an example of like a simple two-dimensional case. And so our ReLU, we're going to get our input to the ReLU, which is going to be a basically you know, W1 X1 plus W2 X2, and it we apply this, so that that defines this this separating hyperplane here, and then we're going to take half of it that's going to be positive,
19:50
and half of it's going to be killed off, and so yes, so you, you know you just, it's whatever the weights happened to be, and where the data happens to be is where these, where these hyperplanes fall, and so,
20:06
so yeah so just throughout the course of training, some of your ReLUs will be in different places, with respect to the data cloud.
20:17
Oh question. [student speaking off mic]
20:23
Yeah.
20:28
So okay, so the question is for the sigmoid we talked about two drawbacks, and one of them was that the neurons can get saturated, so let's go back to the sigmoid here, and the question was this is not the case, when all of your inputs are positive. So when all of your inputs are positive, they're all going to be coming in in this zero plus region here, and so you can still get a saturating neuron, because you see up in this positive region, it also plateaus at one,
21:01
and so when it's when you have large positive values as input you're also going to get the zero gradient, because you have you have a flat slope here.
21:11
Okay.
21:16
Okay, so in practice people also like to initialize ReLUs
21:22
with slightly positive biases, in order to increase the likelihood of it being active at initialization and to get some updates. Right and so this basically just biases towards more ReLUs firing at the beginning, and in practice some say that it helps. Some say that it doesn't. Generally people don't always use this. It's yeah, a lot of times people just initialize it with zero biases still.
21:50
Okay, so now we can look at some modifications on the ReLU that have come out since then, and so one example is this leaky ReLU. And so this looks very similar to the original ReLU, and the only difference is that now instead of being flat in the negative regime, we're going to give a slight negative slope here And so this solves a lot of the problems that we mentioned earlier. Right here we don't have any saturating regime, even in the negative space. It's still very computationally efficient.
22:20
It still converges faster than sigmoid and tanh, very similar to a ReLU. And it doesn't have this dying problem.
22:29
And there's also another example is the parametric rectifier, so PReLU. And so in this case it's just like a leaky ReLU where we again have this sloped region in the negative space, but now this slope in the negative regime is determined through this alpha parameter, so we don't specify, we don't hard-code it. but we treat it as now a parameter that we can backprop into and learn. And so this gives it a little bit more flexibility. And we also have something called
22:59
an Exponential Linear Unit, an ELU, so we have all these different LUs, basically. and this one again, you know, it has all the benefits of the ReLu, but now you're, it is also closer to zero mean outputs.
23:16
So, that's actually an advantage that the leaky ReLU, parametric ReLU, a lot of these they allow you to have your mean closer to zero,
23:27
but compared with the leaky ReLU, instead of it being sloped in the negative regime, here you actually are building back in a negative saturation regime, and there's arguments that basically this allows you to have some more robustness to noise, and you basically get these deactivation states that can be more robust. And you can look at this paper for, there's a lot of kind of more justification for why this is the case. And in a sense this is kind of something
23:58
in between the ReLUs and the leaky ReLUs, where has some of this shape, which the Leaky ReLU does, which gives it closer to zero mean output, but then it also still has some of this more saturating behavior that ReLUs have. A question? [student speaking off mic]
24:20
So, whether this parameter alpha is going to be specific for each neuron. So, I believe it is often specified, but I actually can't remember exactly, so you can look in the paper for exactly, yeah, how this is defined,
24:36
but yeah, so I believe this function is basically very carefully designed in order to have nice desirable properties. Okay, so there's basically all of these kinds of variants on the ReLU. And so you can see that, all of these it's kind of, you can argue that each one may have certain benefits, certain drawbacks in practice. People just want to run experiments all of them, and see empirically what works better, try and justify it, and come up with new ones, but they're all different things
25:06
that are being experimented with.
25:10
And so let's just mention one more. This is Maxout Neuron. So, this one looks a little bit different in that it doesn't have the same form as the others did of taking your basic dot product, and then putting this element-wise nonlinearity in front of it. Instead, it looks like this, this max of W dot product of X plus B,
25:32
and a second set of weights, W2 dot product with X plus B2.
25:38
And so what does this, is this is taking the max of these two functions in a sense.
25:45
And so what it does is it generalizes the ReLU and the leaky ReLu, because you're just you're taking the max over these two, two linear functions. And so what this give us, it's again you're operating in a linear regime. It doesn't saturate and it doesn't die. The problem is that here, you are doubling the number of parameters per neuron. So, each neuron now has this original set of weights, W, but it now has W1 and W2, so you have twice these.
26:18
So in practice, when we look at all of these activation functions, kind of a good general rule of thumb is use ReLU. This is the most standard one that generally just works well. And you know you do want to be careful in general with your learning rates to adjust them based, see how things do. We'll talk more about adjusting learning rates later in this lecture, but you can also try out some of these fancier activation functions, the leaky ReLU, Maxout, ELU,
26:49
but these are generally, they're still kind of more experimental. So, you can see how they work for your problem. You can also try out tanh, but probably some of these ReLU and ReLU variants are going to be better. And in general don't use sigmoid. This is one of the earliest original activation functions, and ReLU and these other variants have generally worked better since then.
27:17
Okay, so now let's talk a little bit about data preprocessing. Right, so the activation function, we design this is part of our network. Now we want to train the network, and we have our input data that we want to start training from.
27:32
So, generally we want to always preprocess the data, and this is something that you've probably seen before in machine learning classes if you taken those. And some standard types of preprocessing are, you take your original data and you want to zero mean them, and then you probably want to also normalize that, so normalized by the standard deviation,
27:56
And so why do we want to do this? For zero centering, you can remember earlier that we talked about when all the inputs are positive, for example, then we get all of our gradients on the weights to be positive, and we get this basically suboptimal optimization. And in general even if it's not all zero or all negative, any sort of bias will still cause this type of problem.
28:24
And so then in terms of normalizing the data, this is basically you want to normalize data typically in the machine learning problems, so that all features are in the same range, and so that they contribute equally. In practice, since for images, which is what we're dealing with in this course here for the most part, we do do the zero centering, but in practice we don't actually normalize the pixel value so much, because generally for images right at each location you already have
28:54
relatively comparable scale and distribution, and so we don't really need to normalize so much, compared to more general machine learning problems, where you might have different features that are very different and of very different scales.
29:11
And in machine learning, you might also see a more complicated things, like PCA or whitening, but again with images, we typically just stick with the zero mean, and we don't do the normalization, and we also don't do some of these more complicated pre-processing. And one reason for this is generally with images we don't really want to take all of our input, let's say pixel values and project this onto a lower dimensional space of new kinds of features that we're dealing with.
29:41
We typically just want to apply convolutional networks spatially and have our spatial structure over the original image. Yeah question. [student speaking off mic]
29:59
So the question is we do this pre-processing in a training phase, do we also do the same kind of thing in the test phase, and the answer is yes. So, let me just move to the next slide here. So, in general on the training phase is where we determine our let's say, mean, and then we apply this exact same mean to the test data. So, we'll normalize by the same empirical mean from the training data. Okay, so to summarize basically for images,
30:30
we typically just do the zero mean pre-processing and we can subtract either the entire mean image. So, from the training data, you compute the mean image, which will be the same size as your, as each image. So, for example 32 by 32 by three, you'll get this array of numbers, and then you subtract that from each image that you're about to pass through the network, and you'll do the same thing at test time for this array that you determined at training time.
31:01
In practice, we can also for some networks, we also do this by just of subtracting a per-channel mean, and so instead of having an entire mean image that were going to zero-center by, we just take the mean by channel, and this is just because it turns out that it was similar enough across the whole image, it didn't make such a big difference to subtract the mean image versus just a per-channel value. And this is easier to just pass around and deal with.
31:29
So, you'll see this as well for example, in a VGG Network, which is a network that came after AlexNet, and we'll talk about that later. Question. [student speaking off mic]
31:45
Okay, so there are two questions. The first is what's a channel, in this case, when we are subtracting a per-channel mean? And this is RGB, so our array, our images are typically for example, 32 by 32 by three. So, width, height, each are 32, and our depth, we have three channels RGB, and so we'll have one mean for the red channel, one mean for a green, one for blue. And then the second, what was your second question? [student speaking off mic]
32:21
Oh.
32:23
Okay, so the question is when we're subtracting the mean image, what is the mean taken over? And the mean is taking over all of your training images. So, you'll take all of your training images and just compute the mean of all of those. Does that make sense? [student speaking off mic]
32:49
Yeah the question is, we do this for the entire training set, once before we start training. We don't do this per batch, and yeah, that's exactly correct. So we just want to have a good sample, an empirical mean that we have. And so if you take it per batch, if you're sampling reasonable batches, it should be basically, you should be getting the same values anyways for the mean, and so it's more efficient and easier just do this once at the beginning. You might not even have to really take it
33:21
over the entire training data. You could also just sample enough training images to get a good estimate of your mean.
33:31
Okay, so any other questions about data preprocessing? Yes. [student speaking off mic] So, the question is does the data preprocessing solve the sigmoid problem? So the data preprocessing is doing zero mean right?
33:48
And we talked about how sigmoid, we want to have zero mean. And so it does solve this for the first layer that we pass it through. So, now our inputs to the first layer of our network is going to be zero mean, but we'll see later on that we're actually going to have this problem come up in much worse and greater form, as we have deep networks. You're going to get a lot of nonzero mean problems later on. And so in this case, this is not going to be sufficient.
34:15
So this only helps at the first layer of your network.
34:22
Okay, so now let's talk about how do we want to initialize the weights of our network? So, we have let's say our standard two layer neural network and we have all of these weights that we want to learn, but we have to start them with some value, right? And then we're going to update them using our gradient updates from there. So first question. What happens when we use an initialization of W equals zero? We just set all of the parameters to be zero. What's the problem with this?
34:56
[student speaking off mic] So sorry, say that again.
35:02
So I heard all the neurons are going to be dead. No updates ever. So not exactly.
35:11
So, part of that is correct in that all the neurons will do the same thing. So, they might not all be dead. Depending on your input value, I mean, you could be in any regime of your neurons, so they might not be dead, but the key thing is that they will all do the same thing. So, since your weights are zero, given an input, every neuron is going to be, have the same operation basically on top of your inputs. And so, since they're all going to output the same thing,
35:40
they're also all going to get the same gradient. And so, because of that, they're all going to update in the same way. And now you're just going to get all neurons that are exactly the same, which is not what you want. You want the neurons to learn different things. And so, that's the problem when you initialize everything equally and there's basically no symmetry breaking here. So, what's the first, yeah question? [student speaking off mic]
36:20
So the question is, because that, because the gradient also depends on our loss, won't one backprop differently compared to the other? So in the last layer, like yes,
36:35
you do have basically some of this, the gradients will get the same, sorry, will get different loss for each specific neuron based on which class it was connected to, but if you look at all the neurons generally throughout your network, like you're going to, you basically have a lot of these neurons that are connected in exactly the same way. They had the same updates and it's basically going to be the problem. Okay, so the first idea that we can have to try and improve upon this
37:04
is to set all of the weights to be small random numbers that we can sample from a distribution. So, in this case, we're going to sample from basically a standard gaussian, but we're going to scale it so that the standard deviation is actually one E negative two, 0.01. And so, just give this many small random weights. And so, this does work okay for small networks, now we've broken the symmetry, but there's going to be problems with deeper networks.
37:36
And so, let's take a look at why this is the case. So, here this is basically an experiment that we can do where let's take a deeper network. So in this case, let's initialize a 10 layer neural network to have 500 neurons in each of these 10 layers. Okay, we'll use tanh nonlinearities in this case and we'll initialize it with small random numbers as we described in the last slide. So here, we're going to basically just initialize this network. We have random data that we're going to take,
38:09
and now let's just pass it through the entire network, and at each layer, look at the statistics of the activations that come out of that layer.
38:23
And so, what we'll see this is probably a little bit hard to read up top, but if we compute the mean and the standard deviations at each layer, well see that at the first layer this is,
38:37
the means are always around zero. There's a funny sound in here.
38:45
Interesting, okay well that was fixed.
38:50
So, if we look at, if we look at the outputs from here, the mean is always going to be around zero, which makes sense. So, if we look here, let's see, if we take this, we looked at the dot product of X with W, and then we took the tanh on linearity, and then we store these values and so, because it tanh is centered around zero, this will make sense, and then the standard deviation however shrinks, and it quickly collapses to zero. So, if we're plotting this, here this second row of plots here
39:26
is showing the mean and standard deviations over time per layer and then in the bottom, the sequence of plots is showing for each of our layers. What's the distribution of the activations that we have? And so, we can see that at the first layer, we still have a reasonable gaussian looking thing. It's a nice distribution. But the problem is that as we multiply by this W, these small numbers at each layer, this quickly shrinks and collapses all of these values,
39:56
as we multiply this over and over again. And so, by the end, we get all of these zeros, which is not what we want. So we get all the activations become zero. And so now let's think about the backwards pass. So, if we do a backward pass, now assuming this was our forward pass and now we want to compute our gradients. So first, what does the gradients look like on the weights?
40:24
Does anyone have a guess?
40:29
So, if we think about this, we have our input values are very small at each layer right, because they've all collapsed at this near zero, and then now each layer, we have our upstream gradient flowing down, and then in order to get the gradient on the weights remember it's our upstream gradient times our local gradient, which for this this dot product were doing W times X. It's just basically going to be X, which is our inputs. So, it's again a similar kind of problem that we saw earlier,
41:01
where now since, so here because X is small, our weights are getting a very small gradient, and they're basically not updating. So, this is a way that you can basically try and think about the effect of gradient flows through your networks. You can always think about what the forward pass is doing, and then think about what's happening as you have gradient flows coming down, and different types of inputs, what the effect of this actually is on our weights and the gradients on them.
41:29
And so also, if now if we think about
41:34
what's the gradient that's going to be flowing back from each layer as we're chaining all these gradients. Alright, so this is going to be the flip thing where we have now the gradient flowing back is our upstream gradient times in this case the local gradient is W on our input X. And so again, because this is the dot product, and so now, actually going backwards at each layer, we're basically doing a multiplication of the upstream gradient by our weights
42:02
in order to get the next gradient flowing downwards.
42:07
And so because here, we're multiplying by W over and over again. You're getting basically the same phenomenon as we had in the forward pass where everything is getting smaller and smaller. And now the gradient, upstream gradients are collapsing to zero as well. Question? [student speaking off mic]
42:31
Yes, I guess upstream and downstream is, can be interpreted differently, depending on if you're going forward and backward, but in this case we're going, we're doing, we're going backwards, right? We're doing back propagation. And so upstream is the gradient flowing, you can think of a flow from your loss, all the way back to your input. And so upstream is what came from what you've already done, flowing you know, down into your current node.
43:00
Right, so we're for flowing downwards, and what we get coming into the node through backprop is coming from upstream.
43:14
Okay, so now let's think about what happens when, you know we saw that this was a problem when our weights were pretty small, right? So, we can think about well, what if we just try and solve this by making our weights big? So, let's sample from this standard gaussian, now with standard deviation one instead of 0.01.
43:35
So what's the problem here?
43:39
Does anyone have a guess?
43:45
If our weights are now all big, and we're passing them, and we're taking these outputs of W times X, and passing them through tanh nonlinearities, remember we were talking about what happens at different values of inputs to tanh, so what's the problem? Okay, so yeah I heard that it's going to be saturated, so that's right. Basically now, because our weights are going to be big, we're going to always be at saturated regimes of either very negative or very positive of the tanh.
44:16
And so in practice, what you're going to get here is now if we look at the distribution of the activations at each of the layers here on the bottom, they're going to be all basically negative one or plus one.
44:31
Right, and so this will have the problem that we talked about with the tanh earlier, when they're saturated, that all the gradients will be zero, and our weights are not updating. So basically, it's really hard to get your weight initialization right. When it's too small they all collapse. When it's too large they saturate. So, there's been some work in trying to figure out well, what's the proper way to initialize these weights. And so, one kind of good rule of thumb that you can use
45:00
is the Xavier initialization. And so this is from this paper by Glorot in 2010. And so what this formula is,
45:11
is if we look at W up here, we can see that we want to initialize them to these, we sample from our standard gaussian, and then we're going to scale by the number of inputs that we have. And you can go through the math, and you can see in the lecture notes as well as in this paper of exactly how this works out, but basically the way we do it is we specify that we want the variance of the input to be the same as a variance of the output, and then if you derive what the weight should be
45:38
you'll get this formula, and intuitively with this kind of means is that if you have a small number of inputs right, then we're going to divide by the smaller number and get larger weights, and we need larger weights, because with small inputs, and you're multiplying each of these by weight, you need a larger weights to get the same larger variance at output, and kind of vice versa for if we have many inputs, then we want smaller weights in order to get the same spread at the output.
46:09
So, you can look at the notes for more details about this. And so basically now, if we want to have a unit gaussian, right as input to each layer, we can use this kind of initialization to at training time, to be able to initialize this, so that there is approximately a unit gaussian at each layer.
46:29
Okay, and so one thing is does assume though is that it is assumed that there's linear activations. and so it assumes that we are in the activation, in the active region of the tanh, for example. And so again, you can look at the notes to really try and understand its derivation, but the problem is that this breaks when now you use something like a ReLU. Right, and so with the ReLU what happens is that, because it's killing half of your units, it's setting approximately half of them
46:59
to zero at each time, it's actually halving the variance that you get out of this. And so now, if you just make the same assumptions as your derivation earlier you won't actually get the right variance coming out, it's going to be too small. And so what you see is again this kind of phenomenon, as the distributions starts collapsing. In this case you get more and more peaked toward zero, and more units deactivated.
47:30
And the way to address this with something that has been pointed out in some papers, which is that you can you can try to account for this with an extra, divided by two. So, now you're basically adjusting for the fact that half the neurons get killed.
47:49
And so you're kind of equivalent input
47:53
has actually half this number of input, and so you just add this divided by two factor in, this works much better, and you can see that the distributions are pretty good throughout all layers of the network.
48:07
And so in practice this is been really important actually, for training these types of little things, to a really pay attention to how your weights are, make a big difference. And so for example, you'll see in some papers that this actually is the difference between the network even training at all and performing well versus nothing happening.
48:33
So, proper initialization is still an active area of research. And so if you're interested in this, you can look at a lot of these papers and resources. A good general rule of thumb is basically use the Xavier Initialization to start with, and then you can also think about some of these other kinds of methods.
48:54
And so now we're going to talk about a related idea to this, so this idea of wanting to keep activations in a gaussian range that we want.
49:03
Right, and so this idea behind what we're going to call batch normalization is, okay we want unit gaussian activations. Let's just make them that way. Let's just force them to be that way. And so how does this work? So, let's consider a batch of activations at some layer. And so now we have all of our activations coming out. If we want to make this unit gaussian, we actually can just do this empirically, right. We can take the mean of the batch that we have so far
49:34
of the current batch, and we can just and the variance, and we can just normalize by this. Right, and so basically, instead of with weight initialization, we're setting this at the start of training so that we try and get it into a good spot that we can have unit gaussians at every layer, and hopefully during training this will preserve this. Now we're going to explicitly make that happen on every forward pass through the network. We're going to make this happen functionally,
50:01
and basically by normalizing by the mean
50:06
and the variance of each neuron, we look at all of the inputs coming into it and calculate the mean and variance for that batch and normalize it by it. And the thing is that this is a, this is just a differentiable function right? If we have our mean and our variance as constants, this is just a sequence of computational operations that we can differentiate and do back prop through this.
50:33
Okay, so just as I was saying earlier right, if we look at our input data, and we think of this as we have N training examples in our current batch, and then each batch has dimension D, we're going to the compute the empirical mean and variance independently for each dimension, so each basically feature element, and we compute this across our batch, our current mini-batch that we have and we normalize by this.
51:06
And so this is usually inserted after fully connected or convolutional layers. We saw that would we were multiplying by W in these layers, which we do over and over again, then we can have this bad scaling effect with each one. And so this basically is able to undo this effect. Right, and since we're basically just scaling by the inputs connected to each neuron, each activation, we can apply this the same way to fully connected convolutional layers, and the only difference is that,
51:37
with convolutional layers, we want to normalize not just across all the training examples, and independently for each each feature dimension, but we actually want to normalize jointly across both all the feature dimensions, all the spatial locations that we have in our activation map, as well as all of the training examples. And we do this, because we want to obey the convolutional property, and we want nearby locations to be normalized the same way, right? And so with a convolutional layer,
52:08
we're basically going to have a one mean and one standard deviation, per activation map that that we have, and we're going to normalize by this across all of the examples in the batch. And so this is something that you guys are going to implement in your next homework. And so, all of these details are explained very clearly in this paper from 2015. And so on this is a very useful, useful technique that you want to use a lot in practice. You want to have these batch normalization layers.
52:38
And so you should read this paper. Go through all of the derivations, and then also go through the derivations of how to compute the gradients with given these,
52:51
this normalization operation.
52:57
Okay, so one thing that I just want to point out is that, it's not clear that, you know, we're doing this batch normalization after every fully connected layer, and it's not clear that we necessarily want a unit gaussian input to these tanh nonlinearities, because what this is doing is this is constraining you to the linear regime of this nonlinearity, and we're not actually, you're trying to basically say, let's not have any of this saturation, but maybe a little bit of this is good, right?
53:25
You you want to be able to control what's, how much saturation that you want to have.
53:32
And so what, the way that we address this when we're doing batch normalization is that we have our normalization operation, but then after that we have this additional squashing and scaling operation. So, we do our normalization. Then we're going to scale by some constant gamma, and then shift by another factor of beta. Right, and so what this actually does is that this allows you to be able to recover the identity function if you wanted to. So, if the network wanted to,
54:04
it could learn your scaling factor gamma to be just your variance. It could learn your beta to be your mean, and in this case you can recover the identity mapping, as if you didn't have batch normalization. And so now you have the flexibility of doing kind of everything in between and making your the network learning how to make your tanh more or less saturated, and how much to do so in order to have,
54:32
to have good training.
54:38
Okay, so just to sort of summarize the batch normalization idea. Right, so given our inputs, we're going to compute our mini-batch mean. So, we do this for every mini-batch that's coming in. We compute our variance. We normalize by the mean and variance, and we have this additional scaling and shifting factor. And so this improves gradient flow through the network. it's also more robust as a result. It works for more range of learning rates, and different kinds of initialization,
55:11
so people have seen that once you put batch normalization in, and it's just easier to train, and so that's why you should do this. And then also when one thing that I just want to point out is that you can also think of this as in a way also doing some regularization. Right and so, because now at the output of each layer, each of these activations, each of these outputs, is an output of both your input X, as well as the other examples in the batch that it happens to be sampled with, right,
55:43
because you're going to normalize each input data by the empirical mean over that batch. So because of that, it's no longer producing deterministic values for a given training example, and it's tying all of these inputs in a batch together. And so this basically, because it's no longer deterministic, kind of jitters your representation of X a little bit, and in a sense, gives some sort of regularization effect.
56:09
Yeah question? [student speaking off camera] The question is gamma and beta are learned parameters, and yes that's the case. [student speaking off mic]
56:28
Yeah, so the question is why do we want to learn this gamma and beta to be able to learn the identity function back, and the reason is because you want to give it the flexibility. Right, so what batch normalization is doing, is it's forcing our data to become this unit gaussian, our inputs to be unit gaussian, but even though in general this is a good idea, it's not always that this is exactly the best thing to do. And we saw in particular for something like a tanh, you might want to control
56:58
some degree of saturation that you have. And so what this does is it gives you the flexibility of doing this exact like unit gaussian normalization, if it wants to, but also learning that maybe in this particular part of the network, maybe that's not the best thing to do. Maybe we want something still in this general idea, but slightly different right, slightly scaled or shifted. And so these parameters just give it that extra flexibility to learn that if it wants to.
57:26
And then yeah, if the the best thing to do is just batch normalization then it'll learn the right parameters for that. Yeah? [student speaking off mic] Yeah, so basically each neuron output.
57:44
So, we have output of a fully connected layer. We have W times X. and so we have the values of each of these outputs, and then we're going to apply batch normalization separately to each of these neurons. Question? [student speaking off mic]
58:10
Yeah, so the question is that for things like reinforcement learning, you might have a really small batch size. How do you deal with this? So in practice, I guess batch normalization has been used a lot for like for standard convolutional neural networks, and there's actually papers on how do we want to do normalization for different kinds of recurrent networks, or you know some of these networks that might also be in reinforcement learning. And so there's different considerations
58:36
that you might want to think of there. And this is still an active area of research. There's papers on this and we might also talk about some of this more later, but for a typical convolutional neural network this generally works fine. And then if you have a smaller batch size, maybe this becomes a little bit less accurate, but you still get kind of the same effect. And you know it's possible also that you could design your mean and variance to be computed maybe over more examples, right,
59:06
and I think in practice usually it's just okay, so you don't see this too much, but this is something that maybe could help if that was a problem. Yeah question? [student speaking off mic]
59:25
So the question, so the question is, if we force the inputs to be gaussian, do we lose the structure?
59:35
So, no in a sense that you can think of like, if you had all your features distributed as a gaussian for example, even if you were just doing data pre-processing, this gaussian is not losing you any structure. All the, it's just shifting and scaling your data into a regime, that works well for the operations that you're going to perform on it. In convolutional layers, you do have some structure, that you want to preserve spatially, right. You want, like if you look at your activation maps,
1:00:06
you want them to relatively all make sense to each other. So, in this case you do want to take that into consideration. And so now, we're going to normalize, find one mean for the entire activation map, so we only find the empirical mean and variance over training examples. And so that's something that you'll be doing in your homework, and also explained in the paper as well. So, you should refer to that. Yes. [student speaking off mic]
1:00:43
So the question is, are we normalizing the weight so that they become gaussian. So, if I understand your question correctly, then the answer is, we're normalizing the inputs to each layer, so we're not changing the weights in this process.
1:01:01
[student speaking off mic]
1:01:15
Yeah, so the question is, once we subtract by the mean and divide by the standard deviation, does this become gaussian, and the answer is yes. So, if you think about the operations that are happening, basically you're shifting by the mean, right. And so this shift up to be zero-centered, and then you're scaling by the standard deviation. This now transforms this into a unit gaussian.
1:01:41
And so if you want to look more into that, I think you can look at, there's a lot of machine learning explanations that go into exactly what this, visualizing with this operation is doing, but yeah this basically takes your data and turns it into a gaussian distribution.
1:02:01
Okay, so yeah question?
1:02:04
[student speaking off mic]
1:02:08
Uh-huh.
1:02:26
So the question is, if we're going to be doing the shift and scale, and learning these is the batch normalization redundant, because you could recover the identity mapping? So in the case that the network learns that identity mapping is always the best, and it learns these parameters, the yeah, there would be no point for batch normalization, but in practice this doesn't happen. So in practice, we will learn this gamma and beta. That's not the same as a identity mapping.
1:02:53
So, it will shift and scale by some amount, but not the amount that's going to give you an identity mapping. And so what you get is you still get this batch normalization effect. Right, so having this identity mapping there, I'm only putting this here to say that in the extreme, it could learn the identity mapping, but in practice it doesn't. Yeah question. [student speaking off mic]
1:03:21
Yeah. [student speaking off mic]
1:03:31
Oh right right.
1:03:34
Yeah, yeah sorry, I was not clear about this, but yeah I think this is related to the other question earlier, that yeah when we're doing this we're actually getting zero mean and unit gaussian, which put this into a nice shape, but it doesn't have to actually be a gaussian. So yeah, I mean ideally, if we're looking at like inputs coming in, as you know, sort of approximately gaussian, we would like it to have this kind of effect, but yeah, in practice it doesn't have to be.
1:04:07
Okay, so ...
1:04:10
Okay, so the last thing I just want to mention about this is that, so at test time, the batch normalization layer,
1:04:17
we now take the empirical mean
1:04:22
and variance from the training data. So, we don't re-compute this at test time. We just estimate this at training time, for example using running averages, and then we're going to use this as at test time. So, we're just going to scale by that.
1:04:40
Okay, so now I'm going to move on to babysitting the learning process. Right, so now we've defined our network architecture, and we'll talk about how do we monitor training, and how do we adjust hyperparameters as we go, to get good learning results?
1:04:58
So as always, so the first step we want to do, is we want to pre-process the data. Right, so we want to zero mean the data as we talked about earlier. Then we want to choose the architecture, and so here we are starting with one hidden layer of 50 neurons, for example, but we've basically we can pick any architecture that we want to start with.
1:05:20
And then the first thing that we want to do is we initialize our network. We do a forward pass through it, and we want to make sure that our loss is reasonable. So, we talked about this several lectures ago, where we have a basically a,
1:05:35
let's say we have a Softmax classifier that we have here. We know what our loss should be, when our weights are small, and we have generally a diffuse distribution. Then we want it to be, the Softmax classifier loss is going to be your negative log likelihood, which if we have 10 classes, it'll be something like negative log of one over 10, which here is around 2.3, and so we want to make sure that our loss is what we expect it to be. So, this is a good sanity check that we want
1:06:08
to always, always do. So, now once we've seen that our original loss is good, now we want to,
1:06:15
so first we want to do this having zero regularization, right. So, when we disable the regularization, now our only loss term is this data loss, which is going to give 2.3 here. And so here, now we want to crank up the regularization, and when we do that, we want to see that our loss goes up, because we've added this additional regularization term. So, this is a good next step that you can do for your sanity check. And then, now we can start training. So, now we start trying to train.
1:06:47
What we do is, a good way to do this is to start up with a very small amount of data, because if you have just a very small training set, you should be able to over fit this very well and get very good training loss on here. And so in this case we want to
1:07:05
turn off our regularization again, and just see if we can make the loss go down to zero.
1:07:12
And so we can see how our loss is changing, as we have all these epochs. We compute our loss at each epoch, and we want to see this go all the way down to zero. Right, and here we can see that also our training accuracy is going all the way up to one, and this makes sense right. If you have a very small number of data, you should be able to over fit this perfectly.
1:07:35
Okay, so now once you've done that, these are all sanity checks. Now you can start really trying to train. So, now you can take your full training data, and now start with a small amount of regularization, and let's first figure out what's a good learning rate. So, learning rate is one of the most important hyperparameters, and it's something that you want to adjust first. So, you want to try some value of learning rate. and here I've tried one E negative six,
1:08:01
and you can see that the loss is barely changing. Right, and so the reason this is barely changing is usually because your learning rate is too small. So when it's too small, your gradient updates are not big enough, and your cost is basically about the same.
1:08:18
Okay so
1:08:21
one thing that I want to point out here, is that we can notice that even though our loss with barely changing, the training and the validation accuracy jumped up to 20% very quickly. And so does anyone have any idea for why this might be the case?
1:08:40
Why, so remember we have a Softmax function, and our loss didn't really change, but our accuracy improved a lot.
1:08:50
Okay, so the reason for this is that here the probabilities are still pretty diffuse, so our loss term is still pretty similar, but when we shift all of these probabilities slightly in the right direction, because we're learning right? Our weights are changing the right direction. Now the accuracy all of a sudden can jump, because we're taking the maximum correct value, and so we're going to get a big jump in accuracy, even though our loss is still relatively diffuse.
1:09:24
Okay, so now if we try another learning rate, now here I'm jumping in the other extreme, picking a very big learning rate, one E negative six. What's happening is that our cost is now giving us NaNs. And, when you have NaNs, what this usually means is that basically your cost exploded. And so, the reason for that is typically that your learning rate was too high.
1:09:49
So, then you can adjust your learning rate down again. Here I can see that we're trying three E to the negative three. The cost is still exploding. So, usually this, the rough range for learning rates that we want to look at is between one E negative three, and one E negative five. And, this is the rough range that we want to be cross-validating in between. So, you want to try out values in this range, and depending on whether your loss is too slow, or too small, or whether it's too large,
1:10:17
adjust it based on this.
1:10:21
And so how do we exactly pick these hyperparameters? Do hyperparameter optimization, and pick the best values of all of these hyperparameters? So, the strategy that we're going to use is for any hyperparameter for example learning rate, is to do cross-validation. So, cross-validation is training on your training set, and then evaluating on a validation set. How well do this hyperparameter do? Something that you guys have already done in your assignment.
1:10:49
And so typically we want to do this in stages. And so, we can do first of course stage, where we pick values pretty spread out apart, and then we learn for only a few epochs. And with only a few epochs. you can already get a pretty good sense of which hyperparameters, which values are good or not, right. You can quickly see that it's a NaN, or you can see that nothing is happening, and you can adjust accordingly. So, typically once you do that,
1:11:16
then you can see what's sort of a pretty good range, and the range that you want to now do finer sampling of values in. And so, this is the second stage, where now you might want to run this for a longer time, and do a finer search over that region. And one tip for detecting explosions like NaNs, you can have in your training loop, right sample some hyperparameter, start training, and then look at your cost at every iteration or every epoch. And if you ever get a cost
1:11:50
that's much larger than your original cost, so for example, something like three times original cost, then you know that this is not heading in the right direction. Right, it's getting very big, very quickly, and you can just break out of your loop, stop this this hyperparameter choice and pick something else. Alright, so an example of this, let's say here we want to run now
1:12:12
course search for five epochs. This is a similar network that we were talking about earlier, and what we can do is we can see all of these validation accuracy that we're getting. And I've put in, highlighted in red the ones that gives better values. And so these are going to be regions that we're going to look into in more detail. And one thing to note is that it's usually better to optimize in log space. And so here instead of sampling, I'd say uniformly
1:12:40
between you know one E to the negative 0.01 and 100, you're going to actually do 10 to the power of some range. Right, and this is because the learning rate is multiplying your gradient update. And so it has these multiplicative effects, and so it makes more sense to consider a range of learning rates that are multiplied or divided by some value, rather than uniformly sampled. So, you want to be dealing with orders of some magnitude here. Okay, so once you find that,
1:13:12
you can then adjust your range. Right get in this case, we have a range of you know, maybe of 10 to the negative four, right, to 10 to the zero power. This is a good range that we want to narrow down into. And so we can do this again, and here we can see that we're getting a relatively good accuracy of 53%. And so this means we're headed in the right direction. The one thing that I want to point out is that here we actually have a problem. And so the problem is that
1:13:45
we can see that our best accuracy here has a learning rate that's about,
1:13:52
you know, all of our good learning rates are in this E to the negative four range. Right, and since the learning rate that we specified was going from 10 to the negative four to 10 to the zero, that means that all the good learning rates, were at the edge of the range that we were sampling. And so this is bad, because this means that we might not have explored our space sufficiently, right. We might actually want to go to 10 to the negative five, or 10 to the negative six.
1:14:21
There might be still better ranges if we continue shifting down. So, you want to make sure that your range kind of has the good values somewhere in the middle, or somewhere where you get a sense that you've hit, you've explored your range fully.
1:14:36
Okay, and so another thing is that we can sample all of our different hyperparameters, using a kind of grid search, right. We can sample for a fixed set of combinations, a fixed set of values for each hyperparameter. Sample in a grid manner over all of these values, but in practice it's actually better to sample from a random layout, so sampling random value of each hyperparameter in a range. And so what you'll get instead is we'll have these two hyper parameters here
1:15:06
that we want to sample from. You'll get samples that look like this right side instead. And the reason for this is that if a function is really sort of more of a function of one variable than another, which is usually true. Usually we have little bit more, a lower effective dimensionality than we actually have. Then you're going to get many more samples of the important variable that you have. You're going to be able to see this shape in this green function that I've drawn on top,
1:15:36
showing where the good values are, compared to if you just did a grid layout where we were only able to sample three values here, and you've missed where were the good regions. Right, and so basically we'll get much more useful signal overall since we have more samples of different values of the important variable. And so, hyperparameters to play with, we've talked about learning rate, things like different types of decay schedules, update types, regularization, also your network architecture,
1:16:08
so the number of hidden units, the depth, all of these are hyperparameters that you can optimize over. And we've talked about some of these, but we'll keep talking about more of these in the next lecture. And so you can think of this as kind of, you know, if you're basically tuning all the knobs right, of some turntable where you're,
1:16:27
you're a neural networks practitioner. You can think of the music that's output is the loss function that you want, and you want to adjust everything appropriately to get the kind of output that you want. Alright, so it's really kind of an art that you're doing.
1:16:42
And in practice, you're going to do a lot of hyperparameter optimization, a lot of cross validation. And so you know, in order to get numbers, people will run cross validation over tons of hyperparameters, monitor all of them, see which ones are doing better, which ones are doing worse. Here we have all these loss curves. Pick the right ones, readjust, and keep going through this process. And so as I mentioned earlier, as you're monitoring each of these loss curves,
1:17:14
learning rate is an important one, but you'll get a sense for how different learning rates, which learning rates are good and bad. So you'll see that if you have a very high exploding one, right, this is your loss explodes, then your learning rate is too high. If it's too kind of linear and too flat, you'll see that it's too low, it's not changing enough. And if you get something that looks like there's a steep change, but then a plateau, this is also an indicator of it being maybe too high,
1:17:42
because in this case, you're taking too large jumps, and you're not able to settle well into your local optimum. And so a good learning rate usually ends up looking something like this, where you have a relatively steep curve, but then it's continuing to go down, and then you might keep adjusting your learning rate from there. And so this is something that you'll see through practice.
1:18:04
Okay and just, I think we're very close to the end, so just one last thing that I want to point out is than in case you ever see learning rate loss curves, where it's ... So if you ever see loss curves where it's flat for a while, and then starts training all of a sudden, a potential reason could be bad initialization. So in this case, your gradients are not really flowing too well the beginning, so nothing's really learning, and then at some point, it just happens to adjust in the right way,
1:18:32
such that it tips over and things just start training right? And so there's a lot of experience at looking at these and see what's wrong that you'll get over time. And so you'll usually want to monitor and visualize your accuracy. If you have a big gap between your training accuracy and your validation accuracy, it usually means that you might have overfitting and you might want to increase your regularization strength. If you have no gap, you might want to increase your model capacity,
1:19:03
because you haven't overfit yet. You could potentially increase it more. And in general, we also want to track the updates, the ratio of our weight updates to our weight magnitudes. We can just take the norm of our parameters that we have to get a sense for how large they are, and when we have our update size, we can also take the norm of that, get a sense for how large that is, and we want this ratio to be somewhere around 0.001. There's a lot of variance in this range,
1:19:34
so you don't have to be exactly on this, but it's just this sense of you don't want your updates to be too large compared to your value or too small, right? You don't want to dominate or to have no effect. And so this is just something that can help debug what might be a problem.
1:19:50
Okay, so in summary, today we've looked at activation functions, data preprocessing, weight initialization, batch norm, babysitting the learning process, and hyperparameter optimization. These are the kind of the takeaways for each that you guys should keep in mind. Use ReLUs, subtract the mean, use Xavier Initialization, use batch norm, and sample hyperparameters randomly. And next time we'll continue to talk about the training neural networks with all these different topics. Thanks.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat