Skip to main content
Loading Explore guides…
Neural Networks · Acedra
Home
Explore
Computer Science
Neural Networks
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
55 segments
Copy
Hide video
0:00
Artificial Intelligence is a broad concept. For example, algorithms used in chess engines pathfinding, and even bots in video games can all be classified as AI. However, these algorithms typically follow a set of predefined rules. A significant part of AI is machine learning, which enables algorithms to learn and create their own rules. At the core of machine learning are Neural Networks, which we will explore in detail in this video.
0:28
In general, neural networks are composed of neurons organized into layers. The first layer is known as the input layer, the final layer is the output layer, and any layers in between are called hidden layers. Each neuron in a layer is connected to every neuron in the next layer. Those connections are called weights. A neural network can be thought of as a machine that takes an input, processes it,
0:54
and produces an output. This concept is similar to mathematical functions, which also take an input and produce an output. In fact, neural networks can be classified as function approximators. For example, the function f(x)=2x+3 is a function that takes a number, multiplies it by 2, adds 3, and gives the result. A neural network can learn to approximate this function. We will delve into how it exactly does that shortly, but for now, I just want you to understand the
1:26
concept of a neural network as a "function" or a machine with input and output. Mathematical functions can have more than one input, and similarly, a neural network can have multiple input neurons. Mathematical functions typically have a single output, but neural networks can have multiple output neurons. When a neural network has multiple outputs, it can be viewed as consisting of multiple functions that share the same inputs. All of this means that neural
1:54
networks can learn anything that can be expressed as a function or a set of functions. For example, language models, like ChatGPT, take words as input, which of course have to be expressed as a numbers, and output the next word. In image recognition, the input could be the color values of every pixel in an image, and the output might be the probability that the image contains a cat. To understand how neural networks work in detail, let's look at a simple neural network with just
2:24
one input neuron and one output neuron, without any hidden layers. These neurons are connected by a weight. Weight is basically just a multiplier. This network would just take a value at the input multiply it by the weight and output it. Since neural networks can be viewed as functions, we can graph them similarly to mathematical functions. The horizontal axis represents the input to the network, and the vertical axis represents the output. With this setup, we
2:53
can approximate functions like f(x)=2x or 3x, or any other multiple of x, by adjusting the weight accordingly. Changing the weight effectively rotates the graph around the origin. However, to express functions like f(x)=x+2, we need to move the graph away from the origin. Adjusting the weight alone won't achieve this, so we introduce a bias. Each neuron, except the input neurons, has a bias. The bias is a value added to the result after the multiplication by the weight.
3:26
By setting the weight to 1 and the bias to 2, we can express the function f(x)=x+2. This simple network can represent any linear function, but it is limited to linear functions only. We can add another input neuron, which will also give us additional weight. Now the output is calculated by multiplying each input value by the corresponding weight, summing them and adding the bias. We can’t show a graph of a function with 2 inputs on a 2D axes,
3:57
so we need another axis. However, it still represents a linear function. We can also add another output. Now, each output is calculated separately. To show this on the graph we would need 4 dimensions, so we will just separate it into 2 different graphs. But they are still just linear functions. Let’s go back to just one input and one output. Next thing we can try is adding a new layer. To calculate the output, we take the input, multiply it by the first weight,
4:28
add the bias, then multiply this result by the second weight, and add the bias of the output neuron. Even with this setup, the output is still a linear function, because combining two linear functions always results in a linear function. Adding more hidden layers or more neurons to each layer also doesn’t help. But let’s just see how we would calculate the output in this neural network. We take the first input multiply it by the corresponding weight, then we add the second
4:57
input multiplied by the corresponding weight, we add a bias, and we have a value of the first neuron in the first hidden layer. Then we do the same for the second neuron. Using these values, we repeat the process for the next layer. And we repeat it until we reach the output layer.
5:19
So, we know that adding additional neurons won’t help us approximate a non-linear function. To approximate any function, we need to introduce something called the activation function. The activation function is applied to a neuron after summing the inputs multiplied by their weights and adding the bias. One of the simplest activation functions is the rectified linear unit, or ReLU. This function takes a number: it outputs 0, if the number is less than or equal
5:48
to 0 or it leaves the number unchanged, if the number is greater than 0. Here's how the graph of this function looks. Now, to calculate the output of this network, we multiply the input by the weight, add the bias, and then apply the activation function. Then we multiply the result by the second weight and add the second bias. In the last layer, we can use the activation function, but we don’t have to. The graph of
6:15
this output would look like this—it’s already not completely linear. By adding more neurons we can create more complex curves, allowing us to approximate more intricate functions. But why do we know that it will work? And why is it called an activation function? Let’s see how this simple network can approximate this function. All weights are currently set to 1 and biases to 0. First, we increase the first weight, which changes the slope of the graph.
6:44
We want this slope in this area of the graph, but after this point, we need a different slope. So, we need to somehow deactivate this slope after that point. To do this, we can increase the second bias, which moves the graph upward. Then, by setting the second weight to negative 1, we achieve a slope of 0 after the desired point. However, this also makes the initial slope negative. To fix this, we negate the first weight. Now the shape looks almost correct,
7:14
but we need to shift everything to the right. We can do this by increasing the first bias. Next, we want to approximate the next part of the curve. So, we repeat the process with a new network. In this part of the graph, we want a smaller slope, so we adjust the first weight. Again, we need a different slope afterward, so we increase the second bias and negate the second weight, as well as the first weight. Finally, we shift the graph to the right by increasing the first bias.
7:47
Now the shape looks good but needs to be moved up. But we are not going to change it anymore. We will just combine both networks into one. We set those weights to 0 and those to 1, and last bias to 0, so the combined network is simply the sum of the two individual networks. As a result, we see that the first part has a different slope than the second part. And this is why it’s called an activation function. It helps us "activate" the neurons in specific regions. In this example,
8:23
the specific values of weights and biases don’t matter; it’s just to show you that with activation functions, we can approximate any curve with enough neurons. And we actually want the computer to find the optimal values for weights and biases. We’ll see how the computer does this in a moment, but first, let’s look at other functions we can use as an activation function. We have the ReLU activation function, which we used earlier. An activation
8:50
function can be even simpler, like this step function, which returns 1 if the input is positive and 0 if it’s not. Another popular activation function is the sigmoid function, which squeezes all values into a range from 0 to 1. There are many different activation functions available, each producing slightly different results. However, the most important thing they all provide is the ability to approximate non-linear functions.
9:16
Now we know how neural networks work, but how do we actually use and train them on our desired data? There are many different ways to train a neural network, but we will focus on supervised learning. In supervised learning, we have some labeled data and we just feed this data to the network and the network learns from the data. To understand training better, let's train a simple neural network to approximate this function. We will choose 50 random points on
9:43
the function from which the network will learn. We will start by setting every weight and every bias to some random value. With those random values the output of the network looks like this. The first step is to measure how well the network is performing. We'll compare the network’s output to each actual data point. For example, looking at this data point, for the input equal to –4.37 the network should output 1.0, but the network outputs –0.91. So,
10:14
we can calculate the error by subtracting 1.0 from -0.91 and we will square the result to not worry about the negative values. We do this for every data point and then take the average of all the errors. We will call this average error the cost of the network. And we will denote it as C. Our goal is to decrease this cost to make the network approximate the function accurately. We can change the network's results, and consequently the cost, by adjusting
10:48
the weights and biases. But how do we determine the best values for these weights and biases? We can start by increasing the first weight by a small amount, let’s say 0.01, and then calculating the cost again. Initially, it might seem like the outputs and errors didn’t change, and the cost stayed the same. However, with more precision, we can see that the cost actually increased slightly. This indicates that increasing the weight raised the cost. Since our goal
11:18
is to decrease the cost, we should actually decrease the weight. We can determine how much the cost changed by subtracting the original cost from the new cost. To make this value more useful, we can express it proportionally to the change in weight by dividing by the amount we increased the weight. Now we will just save this value and go back to the old weight value. Next, we repeat this
11:41
process for the second weight. We increase it by a small amount, calculate the new network outputs, calculate the new errors and the new average cost. We then calculate the proportional change in cost to the change in weight. Now the cost went down so it means that we changed the weight in a good direction. But let’s save this value and go back to the original value of the weight. Now we just
12:05
have to do the same thing for every weight and bias. So, after doing this for every weight and every bias we have the values that tell us how the cost changes proportionally to a small change in every weight and every bias. We should use those values to update the weights and biases, so the network will perform better. So, let’s go back to the first weight. The change in cost proportional
12:28
to the small increase in the weight is equal to 0.14, so if we want to decrease the cost we should also decrease the weight. But by how much? It’s really up to us, but we should generally change the weights by small amounts. So, we will define a value called the learning rate, which we will label as alpha. Let’s say that our learning rate will be equal to 0.02. Now if we want to update the first weight, we just multiply the change in the cost by the learning rate. Keeping in mind
12:58
that we have to decrease the weight we subtract this value from the weight. Now we just have to repeat it for every weight and bias. So, to update the second weight we just subtract -0.23 multiplied by the learning rate from it. This way, we update all weights and biases. Now the network should perform a little better. To see how much the network improved, let’s calculate the outputs, errors and the average cost again. Now we can just repeat this whole process again. With every
13:31
iteration the cost goes down and the network approximates the function a little bit better. And this is how neural networks can learn. However, there is one problem, this method is very slow because we must calculate the cost for every weight and bias, considering every data point and calculating the network’s output each time. While this might be manageable for a small neural network, for larger networks with thousands, millions,
13:57
or even billions of weights, and with much larger data sets, it would take a very long time. The first way to optimize this process is by splitting the data into smaller batches and running each iteration of the algorithm using a different batch. This approach speeds up the process, but we still need to calculate the average cost for every weight and bias. Fortunately, there is an algorithm that allows us to run the network just once for every
14:24
data point and use it to adjust every weight and bias. This algorithm is called backpropagation. In the previous method, we increased the weights by a small amount and observed how that affected the cost. If you know some calculus, you might associate this process with derivatives. Derivatives can help us optimize training the network. But don’t worry if you don’t know how to calculate derivatives—you can still understand the general idea. The derivative of a function f
14:54
with respect to a variable x, in Leibniz notation, is written like this. The derivative tells us the rate of change of the function based on the change in the variable. For example, in this function, here, if we increase x by some amount the function also increases by about the same amount. So, the value of the derivative at this point is approximately 1. Here, the derivative is less than 1 because increasing x by some amount causes the function to increase,
15:26
but by a slightly smaller amount. Here, the derivative is negative because increasing x causes the function to decrease. In our previous method, when we increased the weight by a small amount, we were actually trying to approximate the derivative of the cost with respect to that weight. Now we will try to calculate it directly. Let’s look at this simple network. We'll name the value of a neuron before applying the activation
15:52
function as z, and after applying the activation function as a. Now, let's write the equations to calculate the output of this network. We start with the input, which we will call a0. We multiply a0 by the first weight and add the bias, and we get z1. Then we apply the activation function on z1 to get a1. We multiply the result by the second weight and add the second bias to get z2. Finally, we apply the activation function again to get the output. Next, we calculate the cost.
16:24
Let's say that the true value the network is trying to learn at this input is y. The cost is simply (a2 −y) squared. Now, we want to find the derivative of the cost with respect to the first weight. This derivative will tell us if the cost will increase or decrease when we increase the weight by a small amount. Here, instead of using the normal d to indicate the derivative, we will use a rounded d, because the cost also depends on other weights and biases. The rounded
16:53
d denotes a partial derivative, which is used when a function depends on multiple variables. If you know the chain rule, calculating this derivative won't be a problem. However, looking at the details of how to calculate it will help us better understand backpropagation. We want to calculate the derivative of the cost with respect to the first weight, but this weight is not directly an input to the cost function. According to the chain rule, to find this derivative,
17:20
we first calculate the derivative of z1 with respect to the first weight. Then we multiply it by the derivative of a1 with respect to z1. Next, we multiply that by the derivative of z2 with respect to a1, then by the derivative of a2 with respect to z2, and finally by the derivative of the cost with respect to a2. Calculating each of these simpler derivatives is easier than tackling the derivative of the entire complex function. Moreover, this approach offers another advantage
17:51
we'll see in a moment. But first, let's actually calculate these derivatives. Don’t worry if you don’t know how to calculate derivatives yet; the derivatives here will be generally similar for any neural network. The derivative of z1 with respect to the first weight is just a0. The next derivative, the derivative of a1 with respect to z1, is the derivative of the activation function. In the case of the ReLU function, it is equal to 0 if x is smaller or equal to 0 and 1 if x bigger
18:21
than 0. In this example we are going to use the sigmoid function as the activation function. Its derivative is equal to the output times 1 – the output. So, the derivative of a1 with respect to z1 is just a1 times (1 – a1). The derivative of z2 with respect to a1 is the second weight. The derivative of a2 with respect to z2 is again the derivative of the activation function. Finally,
18:48
the derivative of the cost with respect to a2 is 2 times (a2 − y). So, this gives us the derivative of the cost with respect to the first weight. Now, we can run the network with the data and, for each data point, calculate this derivative by just plugging in the values. If the average derivative comes out to be negative, we increase the weight proportionally to the derivative. If it comes out to be positive, we decrease the weight proportionally to the derivative.
19:16
It might not seem much better than the previous method we used for training the network, because it’s easier to increase the weight by a small number and just calculate the cost of the network than figuring out a formula for each weight and bias. Also adding more layers and neurons will make the formulas for the derivatives increasingly complex. However, let’s see what happens when we calculate the derivative of the cost with respect to the second weight. First,
19:41
we need to see how this weight influences z2, then how z2 influences a2, and finally how a2 influences the cost. This part is the same as in the previous derivative. So, we actually should have calculated the derivative of the cost with respect to the second weight first, and called these values, for example, v2. We would then use v2 to calculate the derivative with respect to the first weight. That is why this algorithm is called backpropagation: we start at
20:10
the end and work our way back. If we add another layer, we calculate the derivative of the cost with respect to the weight of the first layer by multiplying the derivative of z1 with respect to the first weight by the derivative of a1 with respect to z1 and by the derivative of z2 with respect to a1 and then multiplying that by v1, which we will already have if we work backward. Calculating the derivative of the bias is even easier. For example, to calculate the derivative
20:41
of the last bias we just have to change this derivative to the derivative with respect to the bias. And the derivative with respect to the bias is always 1. For the second bias it would be this. When we add more neurons to each layer, the process becomes slightly more complicated, but not by much. Let’s consider a network with 4 layers, where each layer has 2 neurons. We'll need to use some symbols to indicate all the values. The weight connecting the first neuron of the
21:09
first layer to the first neuron of the second layer will be labeled w111. The second weight coming from that neuron will be w121. The weights coming from the second neuron of the first layer will be called w112 and w122. We will use similar naming for the weights in the next layers. The biases will be indicated as such, and the values of the neurons before applying the activation function will be named like this. Similarly, the values of the neurons after applying
21:40
the activation function will be labeled like this. Since this network has 2 output neurons, we also need 2 target values that the function is trying to learn, which will be called y1 and y2. To calculate the total cost, we take the square of a31−y1 and add the square of a32−y2. To get a31, we apply the activation function to z31, and we do the same for a32. To calculate z31, we multiply a21 by w311, add a22 times w312, and then add the bias. We use the same process for z32. For a21,
22:20
we apply the activation function to z21, and we do the same for a22. This process continues back to the first layer. Now let's calculate the derivative of the cost with respect to the weight w311. First, we see how w311 influences z31, multiply that by the derivative of a31 with respect to z31, and then multiply that by the derivative of the cost with respect to a31. So far, this is like the network with just one neuron per layer. Similarly, we calculate the
22:54
derivatives with respect to the other weights in that layer. Now we will call those values v31 and v32. Next, we want to calculate the derivative of the cost with respect to w211. This is a little bit different because after the neuron, the path splits, influencing the cost in two different ways. First, we take the derivative of z21 with respect to w211, multiply that by the derivative of a21 with respect to z21, and then the path splits, because we use a21 to calculate
23:27
z31 and z32. So, we have to sum the derivative of z31 with respect to a21 multiplied by v31 and the derivative of z32 with respect to a21 multiplied by v32. Similarly, we calculate the derivatives with respect to other weights in that layer. Now we can also name those values v21 and v22. And we can do the same for the weights in the first layer. The derivative of the cost with respect to w111 is equal to the derivative of z11 with respect to w111 times the derivative of a11 with
24:03
respect to z11. After that, the path splits in two, so we add the derivative of z21 with respect to a11 times v21 and the derivative of z22 with respect to a11 times v22. The derivatives with respect to other weights follow the same pattern. If we had even more layers, we could call these values v11 and v12 and calculate additional derivatives. To calculate the derivative of the cost with respect to any bias, we use the derivative of the weight coming into the neuron
24:33
and just replace this with the derivative with respect to the bias, which is always 1. Now that we understand how to calculate all these derivatives, let’s look at how the training process actually works. We have some data. Each data point has two input values and two output values. We take the first data point and calculate all the necessary values throughout the network. Once we have these values, we calculate the derivatives, starting from the
25:00
last layer. After calculating the derivatives of the cost with respect to all weights and biases for this data point, we save the values of those derivatives. We repeat this process for another data point and continue for all data points. After we have the values of all derivatives for all data points, we take the average of these derivatives. Now, using those average values, we update the weights and biases in the opposite direction of the derivatives. Which means that
25:30
we subtract the values of the derivatives multiplied by some learning rate we choose. Now, let's test the network to approximate the function we looked at earlier. And because using backpropagation the network can learn much faster, we can add more neurons and the learning will still be fast.
26:01
Next, let's train another network to distinguish between four types of colored dots. This network will have two inputs—the x and y positions of the dot—and four outputs, one for each color. We'll see how the network learns this pattern.
26:28
Finally, let's challenge the network with a more complex pattern to see how well it can adapt and learn.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat