Skip to main content
Loading Explore guides…
Convolutional Neural Networks · Acedra
Home
Explore
Computer Science
Convolutional Neural Networks
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
54 segments
Copy
Hide video
0:00
Hi everybody, in this video, we'll learn all about Convolutional Neural Networks in the context of an Image classification problem. Hi there and Welcome to LearnOpenCV. CNNs were specifically developed to process image data. Here we show the network architecture of one of the earliest CNNs called VGG-16, developed in 2014. At that time this was a state-of-the-art model that was comparable to human-level performance. The diagram shows the general structure of a CNN, which typically
0:31
includes a series of convolutional blocks followed by a number of fully connected layers. The convolutional blocks extract meaningful features from the input image which are passed through the fully connected layers for the classification task. In the previous video on Image classification, we used a densely connected multi-layer Perceptron Network to classify handwritten digits. However MLPs are not well-suited for processing image data for a couple of reasons. One reason is
0:58
that they're not Translation Invariant. What this means for the digit classification task is that the network will react differently depending on where the digit is located within the image frame. So as shown in the example images. When the digits are centered in the image frame, the MLP network will do a fairly good job of identifying the correct digit but if the digits are off center then the predictions are much less reliable as shown in the examples.
1:24
Another reason that MLPs are not well-suited for processing image data is due to the excessive number of parameters in the network which can make the network highly prone to overfitting. This is primarily because image data usually represents a large number of inputs coupled with the fact, the layers in an MLP are all fully connected. So as an example let's consider a fully connected network that takes his input, a color image that's 224x224, which happens to be the
1:53
same size image that's used by the VGG-16 network. However, in an MLP network, this image data needs to be flattened into a one-dimensional vector that contains over 150,000 Elements which represent the raw pixel values in the image. If we then add three fully connected layers with 128 neurons each, then this portion of the network would contain over 300 billion parameters which is extremely
2:19
large and not only would it take a very long time to train such a network, but it would also require a vast amount of data and would be very prone to overfitting on the training dataset.
2:30
Fortunately, there's better ways to process image data. Convolutional Neural Networks were developed to more effectively and efficiently process image data. Let's now take a look at VGG-16 to study the architectural components and the operations associated with individual layers in a CNN. At a high level CNN architectures contain an Upstream Feature extractor followed by a Downstream classifier. Feature extraction segment is sometimes referred to as the Backbone or Body
3:01
of the network and the classifier is sometimes referred to as the head of the network. In this section, we'll introduce all the layer types that form the basis of both network components. To facilitate the discussion, we'll refer to the VGG-16 architecture shown here. The model begins with five convolutional blocks constituting the model's feature extraction segment. A convolutional block is a general term used to describe the sequence of layers in a CNN that are
3:30
often repeatedly used in the feature extractor. The convolutional blocks are followed by three fully connected layers which transform the extracted features into class predictions in the final output layer. Notice that the layers depicted in the architecture each have a Spatial dimension and a Depth dimension. These dimensions represent the shape of the data as it flows through the network.
3:53
The VGG-16 network is specifically designed to accept color images with an input shape of 224 x224x3 where the 3 represents the RGB color channels. As the data passes through the network, the shape of the data is transformed, the spatial dimensions are intentionally reduced, while the depth of the data is increased. The depth of the data is also referred to as the Number of channels. In the sections that follow, we'll learn how this data is transformed using Convolutional
4:25
layers and Pooling layers, two of the most fundamental layers used in CNN architectures.
4:31
Let's now take a closer look at the feature extractor using a different diagram to visually depict the layers in the network. The specific layer is within a convolutional block can vary depending on the architecture but at the most basic level a convolutional block typically contains one or more 2D convolutional layers followed by a pooling layer. Other layers are also sometimes incorporated, but we'll focus on these two layer types to keep things simple. With this
5:00
structure in mind, we're now ready to introduce the general concept for convolutional layers, which can be thought of as the eyes of a CNN. The neurons in a convolutional layer look for specific features. At the most basic level, input to the very first convolutional layer is the input image which is typically either a grayscale image, a single channel or a color image. With three channels
5:24
in the case of VGG-16, the input is a color image with the shape of 224 x224x3. Here we depict a high-level view of a convolutional layer. Convolutional layers use filters to process the input data, the filters moved across the input, and at each filter location a convolution operation is performed which produces a single number. This value is then passed through an activation function and the output from the activation function populates the
5:54
corresponding entry in the output also known as an Activation Map with the shape of 224x224x1.
6:03
You can think of an activation map as containing a summary of meaningful features from the input via the convolution process followed by a transformation through the activation function.
6:16
Before we can describe convolutional layers in more detail, we need to first take a small detour to explain how the convolution operation is performed. In a convolutional layer, a small filter also referred to as a Kernel usually, 3x3 or sometimes 5x5 is used to process the input data. In this example, we show how a 6x6 input is convolved with a 3X3 kernel. Here we use a well-known kernel often used in image processing called a Sobel Kernel which is designed to detect
6:50
vertical edges. The elements in this kernel are fixed however, it's important to note that in CNNs the elements in the kernel are weights that are learned by the network during training. The convolution operation consists of placing the filter over a portion of the input and multiplying the elements of the filter with the corresponding elements of the input. The resulting value is a single number representing the output of the convolution operation for a given filter location.
7:19
Sliding the filter one pixel at a time corresponds to a stride of one. The filter location is shown in dark blue and this region is also referred to as the Receptive Field. Notice that the center of the filter is indicated in green. The convolution operation performed at each filter location is simply the dot product of the filter values with the corresponding values in the receptive field in the input data. A sample calculation is provided for the first two filter
7:49
locations so you can confirm your understanding of the operation. Notice that at each filter location the operation produces a single number placed at the corresponding location in the output. In other words, the location and the output corresponds to the center of the receptive field overlaid on the input slice. Notice that the output from the convolution operation has a smaller spatial size than the input slice. This has to do with how the filter is placed
8:19
over the input data so that it does not extend beyond the boundary of the input. However padding techniques are often used to pad the input image so that the output is the same size as the input. Using a stride larger than one will also reduce the size of the output from the convolution operation. However, in the remainder of this video for simplicity, we'll assume a padding configuration that maintains the spatial size of the data in the convolutional layers.
8:47
Before we continue, we wanted to give you a more concrete example of how specific kernels can detect basic structures in an image. This is a really nice website that allows you to interactively experiment with various kernels using a pixelated input image so that you can appreciate the computations involved in the convolution operation and the associated output shown to the right. As you can see when the kernel moves from left to right across the image, it's
9:15
able to detect vertical edges in the input image. Other kernels can also be used to detect different types of edges for example horizontal edges or diagonal edges, but the important point here is that CNNs generalize this concept since the kernel weights are learned during the training process and therefore CNNs have the ability to detect many different types of features automatically.
9:38
In other words, the network is incentivized to come up with kernels that extract meaningful features whatever those may be. Recall from the discussion earlier, the output of a convolution operation is passed through an activation function to produce what are known as Activation Maps. Convolutional layers typically contain many filters which means that each convolutional layer produces multiple Activation Maps. The operations associated with convolutional
10:07
layers can be confusing for a couple of reasons. One reason is that it's difficult to visually depict all these details and the other is getting a handle on the terminology.
10:17
So if you have any doubts about the material covered thus far, we've included this Summary of Key points as a reference. So with these ideas in mind let's dive into a simple example.
10:29
Suppose the input image has a size of 224x224x3, there's no specific requirement for the spatial size of a filter but it must have a depth of three to match the depth of the input image. In the example below, we use a single filter of size 3x3x3. Here we've chosen to use just a single filter and therefore the filter contains three kernels where each kernel has nine trainable weights, so there's a total of 27 trainable weights in this filter plus a single bias term for a total
11:00
of 28 trainable parameters. Because we've chosen just a single filter, the depth of our output is 1, which means we produce just a single channel activation map as shown. When we convolve this single three-channel filter with the three-channel input the convolution operation is performed for each channel separately and then the weighted sum of each of the three channels plus a bias term is
11:24
passed through an activation function whose output is represented as a single number in the output activation map shown in blue. So just to summarize, the number of channels in a filter must match the number of channels in the input and the number of filters in a convolutional layer which is a design choice dictates the number of activation maps that are produced by the convolutional layer.
11:47
Let's now proceed to another example that represents a more general case. First notice that the input is shown as having a depth of three, but this doesn't necessarily correspond to color channels. It just means that we have an input tensor that has three channels. Keep in mind that when we refer to the input we don't necessarily mean the input to the neural network but rather the input to this convolutional layer which could
12:13
represent the output from a previous layer on the network, such as a series of Activation Maps. In this case we've chosen to use two filters and because the depth of the input is 3 that means each filter must also have a depth of three which means that there's a total of 54 trainable weights because each filter contains 27 weights and then we have one bias term for each filter so we have
12:37
a total of 56 trainable parameters. Before we move on to pulling layers, it's worth discussing some of the intuition associated with filters. You can think of each filter in a convolutional layer as learning something special and unique about the input data to that layer, whether that represents the input to the network or the output from a previous layer in the network. Remember that each
13:02
filter contains weights that are learned during the training process, and because the weights are initialized to random values each filter will learn to extract different features from the input. Here we show a single-channel input with various linear components and to be more precise the lines shown in the input would normally represent edge structures in the image. In this example, we have a convolutional layer with three filters, and we're showing that each
13:29
filter learns to detect different types of edges. In the output Activation Maps, we're emphasizing the highly activated neurons associated with each filter. So for example, the third filter learns to detect diagonal lines and so the third activation map will have highly activated neurons that correspond to that edge structure in the input data. This general idea can then be extended as the data in a CNN is processed through several layers. It turns out that CNNs are able
14:00
to learn hierarchical features that are based on lower level elements like edges, corners and arcs. This starts to become very difficult to visualize across many layers but a paper published in 2013 developed a novel visualization technique that gives insight into how CNNs are able to learn hierarchical features. In the diagram, we showed two examples from the original research paper to illustrate that filters in the first layer learn to detect basic structural elements like
14:30
edges and color blobs while layers deeper in the network are able to leverage the data from previous layers to detect more complex compositional structures like the wheels of a car. So we hope that gives you a better feel for how CNNs use features to characterize image content.
14:48
At this point, we're now ready to study pooling layers. In most CNN architectures it's typical that the spatial dimensions of the data is reduced periodically via pooling layers.
15:00
Pooling layers are typically used after a series of convolutional layers to reduce the spatial size of the activation maps of a preceding convolutional layer. You can think of this as a way to summarize the features from an activation map. Using a pool-in layer will therefore reduce the number of parameters in the network because the input size to subsequent layers is reduced. This is a desirable effect because the computations required for training are
15:26
also reduced and using fewer parameters often helps mitigate the effects of overfitting. Pooling is a form of downsizing that uses a 2D sliding filter that passes over the input slice according to a configurable parameter called the Stride. The Stride is the number of pixels that the filter moves across the input slice from one position to the next between pooling operations.
15:50
There are two types of pooling operations Average pooling and Max pooling, however Max pulling is the most common. For any given filter location the corresponding values in the input slice are passed through a Max operation and the maximum value is recorded in the output. As shown in the example, we have a 4x4 input slice a two by two filter and a stride of two. The corresponding output is, therefore, a two by two downsize representation of the input slice.
16:21
Contrary to convolution of filters, the filter in a pooling layer has no trainable parameters. The filter simply specifies the size of the window to use for performing the max operation.
16:33
Now that we've covered all the basic elements, let's now take a look at what a very simple convolutional block looks like at the beginning of a network.
16:42
For simplicity we show a single convolutional layer containing a single filter. The input to the layer is the input image. In the case of vgg-16 this is a color image indicated by the three RGB channels with a spatial size of 224x224. After the convolutional layer, a Max pooling layer is added to reduce the spatial dimension of the activation map to a size of 112x112. The only difference between this diagram and the convolutional blocks used in CNN architectures
17:15
like BGG-16 is that there are usually two or three consecutive convolutional layers followed by a Max pooling layer and the convolutional layers usually contain at least 32 or more filters. At this point, we're now ready to talk about the transition from the Feature extractor to the Classifier. As discussed earlier the fully connected dense layers in a CNN architecture transform extracted features in the class probabilities.
17:45
In the case of VGG-16, the output from the last convolutional block Conv-5 contains a series of activation maps with the shape 7x7x512. For reference, we've added a block diagram view of the network layers indicating the number of channels at key points in the architecture.
18:06
Before the data from the last convolutional layer in the feature extractor can flow through the classifier, it needs to be flattened to a one-dimensional vector of length 25,088. After flattening this one-dimensional layer is then fully connected to FC-6, as shown in the diagram.
18:27
Let's try to give some intuition for why the fully connected layers are needed and well suited for the classification task. The activation maps from a final convolutional layer of a trans-CNN network represent meaningful information regarding the content of a particular image. Connecting the flattened output from the last convolutional layer in a fully connected manner to the classifier allows the classifier to consider information from the entire
18:54
image. Remember that each spatial location in a feature map has a spatial relationship in the original input image and just to note that the flattening operation indicated here is required for processing the Activation Maps through the classifier but it doesn't alter the original spatial interpretation of the data, it's just a repacking of the data for processing purposes.
19:17
To make this all a little more concrete, let's assume we have a trained model, which means all the weights in the network are fixed. Now consider an input image of a sports car and an elephant. These images have very different content and therefore as the data for each image is processed through the network, the resulting feature maps at the last convolutional layer will look much different.
19:41
It's important to remember that all the weights in the network are now fixed. In the feature extractor the network has learned to detect a wide range of features across all the images in the training dataset. When these activation maps are processed through the classifier, it's the combination of the feature maps and the trained weights in the classifier that lead to higher activations for the
20:03
specific output neurons associated with each input image type. The notional dotted lines between the fully connected layers represent pathways leading to the highest activations in the output neurons. In other words, large weights coupled with high activations at each layer. In reality, there's over 100 million connections between the flattened activation maps and the first fully connected
20:27
layer in the classifier, so this is a strictly a notional graphic to illustrate the basic concept. For anything larger than a toy problem we can't possibly internalize the thousands of mathematical operations and transformations that lead to the successful mapping, but at a high level we understand that the weights in the network were trained in a principled way to minimize a loss function to learn meaningful features and to fire the appropriate neuron in
20:54
the output layer. This is the essence of how such a network maps features to class probabilities.
21:01
We covered a lot of material in this video and have summarized the key points below for your reference. If you'd like to learn more about Deep Learning we offer courses that cover this topic in greater depth at the link below. We hope you found this helpful, and if so, please LIKE and SHARE the video and SUBSCRIBE to our channel to get more content in Computer Vision, Deep Learning, and AI.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat