Skip to main content
Loading Explore guides…
Sensor Fusion for Autonomous Vehicles · Acedra
Home
Explore
Engineering
Sensor Fusion for Autonomous Vehicles
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
77 segments
Copy
Hide video
0:00
Hello. I'm Robert Laganiere. And welcome to my presentation on Sensor Fusion for Autonomous Vehicles. So together we will look at different sensor use in the automotive system. We will overview the different strategy used to combine them and describe some artificial intelligence method in order to perform object detection from sensor fusion. But first, just a few words about myself I'm a professor at the University of Ottawa, a researcher in computer vision in artificial intelligence.
0:37
We think that as an object detection, tracking, recognition, as well as embedded vision and convolutional neural network. I'm also involved with a number of startup companies, in particular, Tempo Analytic, which develop solutions for customer tracking and retail application. And Sensor Cortek that develop artificial intelligence system for autonomous vehicle and for all type of assistance application. Since this in this presentation, we want to talk about the sensor of fusion
1:17
in the context of autonomous vehicle. So I did a quick search for a different article to see where the industry currently stands for the development of driverless vehicle. And as you can see from these headlines, the automaker seems to be very optimistic for a quick development of fully autonomous vehicle. But this article are from more than a year ago. If we look at the 2020 article, the message is quite different. And the industry doesn't seem to be any more that optimistic in terms
2:03
of the development of the autonomous vehicle. Why is that? Well, because even if today, from the recent advances in artificial intelligence, we can say that driving a vehicle on highway on a sunny day can be considered as a solved problem. In general, we are far from having a really autonomous vehicles. If you think of a vehicle without a steering wheel being able to drive anywhere, anytime under any weather condition, the problem is much more difficult. In fact,
2:52
this is basically a perception problem, a very, very difficult perception problem. So one of the solution in order to improve the perception of autonomous vehicle, in order to make them more autonomous, well, is to add multiple sensor to the vehicle so use and to use multiple type of sensor. So the main ones that are currently used are the following, the lidar or laser scanner. So this is a sensor based on laser that is able to measure distance from the vehicle to the driving--
3:42
the objects of the driving scene. You have also the camera, which is a well-known sensor that capture the visible light, or in some cases, can be extended to infrared and night vision. And you have the radar, which is also an active sensor and that use waveform in order to detect a moving target around the vehicle. If we look at each of these sensor, we can see that they each has their own advantages and disadvantages. For example, the camera is a passive sensor
4:31
so it doesn't have to emit a signal in order to read the environment. It's a relatively low power sensor that is very strong at classifying, recognizing, detecting objects as long as these one are visible. As I mentioned, you can also extend the visibility of objects beyond the visible spectrum so you can use, for example, a thermal camera. And there's no interference when you use multiple camera. Camera of relatively high resolution. And also the research in the artificial intelligence
5:21
research with camera images is very, very advanced. However, camera depends on visibility so and so on on good lighting condition. They are very affected by shadows, reflection, heavy sun, and also very important, they get dirty very easy. And the camera cannot provide direct 3D information, unless you use a stereo system of camera.
6:01
The lidar, the main benefit of these laser scanner is that they provide 3D information. They can work day and night. It's a very accurate sensor that can be of quite high resolution if you ready to pay the price. The research in artificial intelligence using lidar is also very well-advanced, however, this is an expensive sensor. And it does not provide any information about the appearance of object, which can make a classification and recognition more difficult. They can become ineffective
6:47
on the rain and fog because of the reflection you get from the water drops. And it also contains a mechanical rotating parts which can make the sensor less reliable. Radar finally, it's the only sensor that can capture direction, distance and velocity directly. It's an inexpensive sensor that can work days and nights. It's also the one that is probably the most resistant to adverse weather condition. It could be relatively a long range if you are ready to pay the price in terms of batteries.
7:50
And it's a reliable, proven technology because it has been used for years. And some relatively basic solid state security that can be used to build radar sensors. However, it's a very noisy sensor with very limited classification capability. It has a poor little resolution even if the newest radar are improving a lot in terms of the definition they can get. Because of that, they cannot really detect a small object at least from far, and the researcher in artificial
8:39
intelligence on the subject is just stopping. So as we can see, they all the opposing counts. So there's really no one sense of that you can use that would solve all problems or perception problem in a vehicle.
9:01
This is another illustration of this, where you have those pie representation that show how well, according to different feature, each sensor is doing. As you can see, if you read from the presentation on the right, for example, if you look at the velocity detection dimension, we can see that radar are very good at that while velocity is very difficult to estimate from camera. Camera are very good, for example, for traffic sign classification.
9:41
While it's certainly not the case with RADAR or LiDAR. Really, since we cannot get all feature from one sensor, the solution is really to perform sensor fusion. That is, to combine the properties of each of the sensor in order to get what we hope will be the perfect sensor. Just a side note, on the fact that sensor fusion should not be confused with system redundancy. This is of very ordinance here. It's a very important concept for a high-reliability system where you want to take an action only
10:30
if you're certain that you have detected some particular event. For example, if you have an auto braking system you might have to have two or three independent systems that detect obstacle in front of you. And only if at least two of these systems confirm that there is an obstacle in front of the vehicle, and only then, you will apply the brake on the vehicle. In that case, it's really, you're looking really for an intersection between the decision of all sensor, of all system.
11:10
In the case of sensor fusion, what you want to achieve is more the union of all the property of the sensors.
11:22
Now, before we go into the details of the sensor fusion concept, let's have a look or let's recap how we can proceed to object detection or sensor AI processing for just a single sensor. In general, what you would do, you would read the input of the sensor, the raw input provided by the sensor. You will process it using a deep convolutional network. The role of this convolutional network is really to build some intermediate representation that will extract the most relevant information
12:08
from the raw data of the sensor. Not that I mention here, the raw data. But as we will see in a minute, you can also pre-process the input from the sensor in order to build a more relevant representation of the sensor data. You will process that signal in order to obtain, at the end, what we call a feature map. Which is really an embedded representation in the form of a big matrix of numbers that really contain high level, mid-level representation of the scene that the sensor captures.
13:06
Really, for example, if there is a wrong object, well some part of that matrix will be able to confirm that if there is some lines or some structure that looked like a car, there will also be other parts of the presentation that will confirm that. Such that by reading that big matrix of number and passing it to a classifier, which is usually another neural network, call a fully-connected network. Then you will build your classifier. And you will be able to output the detection result
13:50
in the form of what type of object we are observing, where is the object? What is the shape of the object? The size of the object, et cetera. That's really how you would proceed if you use a convolutional network in an artificial intelligence context for object detection.
14:16
If we continue on our recap, if you want to train such a system, and really what you need to do is to be able to build, to have at your disposal, I should say, an annotated data set that contain images with the detection you would like to obtain. So this is really that kind of data set that you need in order to train your system. Basically, the idea is that you input one of these image to your system. You look at the detection results. If the detection results are not the one you are expecting,
15:07
you need to modify or to adjust the different ways inside your deconvolutional network in order to correct this error. This is usually done by defining what we call a loss function. That is basically low if the error is small and high if the error is more important. By looking at the value of this error function, then you back propagate this error into your network in order to incrementally modify your weight. This is basically how you would proceed
15:49
to train a neural network in artificial intelligence. As an example, I can show you the result, the kind of detection result we get for a LiDAR system. This is just a quick video that showed the detection results. We illustrate the detection results are vehicle on the camera image. But not that in this case, the detection are just done using the LiDAR information that is below. That is made basically of 3D points. That's how we perform the detection. What you see are 3D bonding docks
16:41
showing the location of the vehicle in the scene. In this example, what we have done is we built a representation for the LiDAR points in order to process them using a convolutional network. Because basically, the raw input a LiDAR system is a very large set of 3D points, which is not very suitable for the convolution operation use inside a convolutional network. In that particular case, the intermediate representation we build is Voxel based. Which means that we superimpose a grid of 3D boxes
17:34
onto the cloud of 3D points and for each, inside each of these voxel, we count the number of points inside this box. We can count. We can just have a binary value, saying yes, does this occupy or not? We can look at the power response obtained at this particular location, et cetera. So there's a different option that one can use. But this is not the only representation that can be used to process a LiDAR signal. You can also build what we call a bird's-eye view.
18:18
Which is basically just the projection of all the 3D points onto the ground plane. You can also build what we call a height map. Which is basically a view from the vehicle off the LiDAR result. In that case, each pixel represent the height of that particular point, with respect to the ground. Or you can build a range map where in this case, the image, each pixel of the image represent the distance from that point to the vehicle. All these representations can be used in order
19:01
to perform detection. And it's really a matter of making a decision when you design the overall network, and look at the performance you get for each representation. Since we talk about data representation, what about RADAR? RADAR is a very old technology that use a well-defined sequence of steps in order to process that kind of signal. Obviously, if you want to build a convolutional network using RADAR signal, you can follow, more or less, the same steps
19:41
or not, depending on the representation you want to use. But basically, a RADAR will emit a waveform and will listen at the echo of these waveform. You receive the signal from the antenna of the RADAR. You convert them into a digital signal. After that, you must apply a couple of fully transform in order to obtain the two different signal that are very useful in the context of RADAR. The first one is the range Doppler. That is basically a representation of the distance
20:28
and the speed of the different target that you see around the vehicle. So this is one particular representation. Another one is the range angle signal that you can get. This is also called the direction of a rival system, the signal, the DOA. It's basically, for each angle in front of you, what is the distance of the target that you can see in that particular direction. From these signals, the classic way of processing the RADAR is to perform an additional step in order
21:15
to extract the important points in the image in the signal, in the scene. This will create a sort of a point cloud. A bit like you are when you use LiDAR. From that point cloud, after that, you might want to use object tracking in order to extract the main object of the scene. For example, if you have a blind spot monitoring system in your car, that is based on RADAR. The object tracking module will be able to turn on a light just to notify the driver that there is a vehicle in the blind spot
22:01
whenever it is detected. If you want to build a AI system that process the RADAR signals, you can use any of these, point cloud, DOA, or range-Doppler presentation in order to perform your object detection. OK. Now that we learn about all the sensor and the different properties, let's have a look at multi-sensor fusion, in the context of artificial intelligence. What we want to do is to be able to train a network with input representation from more than one sensor, more than one sensor type.
22:45
So in order to be able to do so, what you must have is a set of sensors that are all calibrated and registered one with respect to the other. Which means that whenever they provide information about localization of object, we want to make sure that we obtain that information within a common reference frame. The sensor must also be synchronized in some way. Or at least, they have to use a common time reference such that whenever one sensor capture a frame,
23:25
we know exactly the time at which this occurred. If we have that, we will be able to build a multi-branch neural network, one branch for each sensor. And this network will output a single decision at some point. Which means that we must find a way to combine the information coming from these multiple branches.
23:59
There are different multi-sensor fusion strategy that can be used in order to perform the sensor fusion. The simplest one is to use what we call early fusion. In that case, all we have to do is to combine the signal coming from a multiple sensor. And then build a network that will produce a predictor that will output detection results.
24:35
The other strategy is delayed fusion. In that case, you process independently each signal coming from the different sensor. Then, at the end you combine them in order to produce a detection result. In between these two strategy, you have the mid-level fusion that would first build an intermediate representation from each sensor. These sensor representations will be combined, in order to finally produce a detection result. We will also look at the sequential fusion approach.
25:17
In that case, what you do is that you use the information coming from each sensor, but one after the other. So the information provided from one sensor will help the second sensor to make a better decision. But let's look in more detail at these different approaches. The early fusion approach. The idea is very simple. you will try to combine all the data coming from the different sensor. You are trying to combine them into one input data. You will have to create that hybrid data, data input.
26:09
In order to be able to do so, ideally, you have to signal data of the same size that have the same resolution. If it's not the case, you will have to pre-process some of your RADAR data, of your sensor data, in order to be able to easily combine them. Also, this data, it should be similar in nature, in nature. A simple example of this case is the thermal versus the visible camera. These are two sensors that are quite easy to combine. One pixel in the thermal image correspond to the same scene
27:03
element in the visual image so they are very similar, these two type of signal. So it's a very good candidate for early fusion. Once you have identified your two sensor and the way to combine them. After that, building your network and training your network, it's really like using a single sensor system. Again, in the case of our thermal plus visible image. So the input becomes simply a multichannel image, where the first few channel contain the information about the visible image.
27:51
And probably the last channel will contain the thermal information. This becomes just one signal that is processed, as if you have just one sensor.
28:10
This is for early fusion. As I mentioned, ideally, the input data should be similar in nature. Which is not the case if you, for example, you would like to combine a visible camera with a bird's-eye view representation that you get from a LiDAR sensor. In that case, it is more difficult to try to combine them. They don't have the same resolution, the same size. Moreover, if you look at one pixel in the image, it is hard to predict where, in the bird's-eye view image
28:51
of the LiDAR, where this pixel of the object visible from that pixel location will be represented in the LiDAR sensor. For that reason, you might want to look at the late fusion approach. In that case, what you have to do is to simply process each of the sensor independently. At the end, you combine them into one big feature maps. If we look at the structure of such a network, you really have two independent convolutional network. Each of them are building its own feature map.
29:38
And these two feature map are then combined to create one integrated feature map. That will contain information from both sensors. Now, what strategy can we use in order to combine these feature maps? You can use very simple functions such as simply summing the result of each of the entries, after that big matrix. You can use a max function that will only keep the response from the sensor, which has the largest, the highest value. Or you can use more complex scheme
30:29
that consists in just concatenating these two feature on map and use a filter in order to produce an integrated feature map. All these strategy will produce one feature map at the end. This is this integrated feature map that will be used by the classifier in order to win to output the information about the detection result. In more detail, this is a late fusion network that we are currently developing. As you can see, in that case, one of the network process
31:20
take image while the other one process a point cloud in the form of voxels. The idea here is that the LiDAR is very good at localizing the 3D object in the scene. We use that branch in order to produce preliminary detection result. Then we used the appearance of the object visible from the camera in order to confirm if yes or no, we have a detection from that location, detected by the LiDAR. This information is combined in order to get the classification at the end.
32:21
With such an approach, and these are just preliminary result, we have been able to improve the accuracy. Let's just have a look at the last column in that case. We have been able to improve the accuracy in the detection. If we compare just using LiDAR versus using both camera and LiDAR, we obtain a boost of 13%. But as you can see, this is more significant in terms of for pedestrian than it is for car. That's because pedestrians are smaller objects that
33:04
are more difficult to detect using point cloud. While a car are bigger objects that can be seen clearly just using LiDAR. This is why, in that case, it's mainly the pedestrian detection that gets improved. If we look at some specific result to illustrate that, we can see that bottom are the result of the LiDAR-only detection. The LiDAR system detects two pedestrian, in that case. But if we look at the image, we can see that one of the pedestrian detected is not the pedestrian, in fact.
33:48
This is just a 3D structure that look like a pedestrian, from the LiDAR point of view. By using the fusion network we have been able to clean that and just detect the right pedestrian. By the way, that pedestrian which is quite far away might have been missed by the camera only. But since it was clearly visible from the LiDAR, it has been detected correctly. You have exactly the same example here, where you have a false pedestrian that has been detected by the LiDAR.
34:27
But that pedestrian was just a 3D structure again, that looks like a pedestrian and was rejected by the fusion network.
34:39
In general, when you work with a detection system, and you want to determine the performance, you would use two main metrics. One is the precision. And the precision is to answer the question are we detecting the right thing. When we say that this is a pedestrian, is it really a pedestrian? The recall metric is are we detecting everything? Whenever there is a pedestrian in the scene, are we sure that we will be able to detect it? Whenever you design a detection system,
35:23
you will get either a very high precision but maybe a lower recall. Or the opposite, a very high recall and a lower precision. It's very difficult to get both. If you look at the fusion strategy, usually the early fusion strategy will get high precision. Because you're processing a very rich sensor that is made of the concatenation of the information from many different sensors. From that information, whenever you detect something, you're pretty sure that you detect the right thing.
36:05
While if you're using a late fusion approach, very often you will get a high recall. Because you independently detect object form the different sensor. So whenever there is something you have a good chance that one of the sensor will be able to see it. That way, you will get a very high recall. But because you process each sensor independently, you might not always be as accurate as you wish. But this is just a very general statement. But that's, more or less, what you
36:45
could expect from these two different central fusion strategies. Now, let's have a look at the third approach, which is the mid-level fusion. Which could be, in a sense, considered as the most optimal configuration? Indeed, what you will do in that case, you will first build feature map from each of the sensors. So you will have two branches or a number of branches corresponding to the number of sensor you use. These branches will create intermediate representation.
37:31
That will then be combined at the mid level of the network. And a new branch will then process that combined system signal, in order to generate, at the end, a prediction about the detected object.
37:54
Because you have now to combine a more complex representation, very often you will want to use a more complex fusion mechanism. Mid-level fusion is such a representation that it's more complex, a strategy that is more complex in order to combine your system. But you can also use more basic ones. I invite you to look at the reference there in order to learn more about these mechanism. That seems to be really the most optimal approach. But it turns out that mid-level representation,
38:44
mid-level fusion networks are much more difficult to train. And this can be explained by the following description. Basically, as I told you, when you will have to train this system, whenever the end result detection is wrong, you will have to propagate the error from the top of the network to the lower layers, in order to adjust the weight for the network to converge to a network of higher accuracy. However, when you have two branches like that and you have to propagate the error,
39:32
at the point of intersection when you perform your propagation, you won't know exactly. And you won't know exactly where you should go. And is the error due to the camera branch? Or is it, in that case, due to the LiDAR branch? For this reason, is much more difficult to correctly train your network.
40:00
In practice, you can use many different strategy in order to obtain a strong mid-level fusion network. But this is, in practice, quite complex. I just see illustrate here one of the network that we have designed.
40:27
This show you the different element that we have to add in order to have our network being able to learn and to process efficiently, in that case, LiDAR and image data. We incorporate different strategies. One of them, I just will talk about just one of them, is the fact that we use an independent branch in order to process the LiDAR data and the camera data into using an early fusion strategy. This allow us to basically pre-train that branch before training the fusion network.
41:20
This way, after that, we propagate the error. It's mainly the older network that gets corrected in order to obtain a more performing fusion network. Again, I invite you to look at the different references to learn more about that strategy.
41:45
The last type of fusion, the fusion approach, is the sequential fusion.
41:56
This is a fusion. But in the sense that you will use multiple sensor in order to get your detection result. But in this case, each sensor will be used one after the other.
42:15
The idea is really to exploit the strength of each network to guide the decision of the next one. For example, we know that a visual camera is very good at detection while LiDAR is good at 3D localization. There's a way to benefit from these different properties. But one of the risk is that since we're using each sensor one after the other, we might get an accumulation of errors. Whenever one sensor output a wrong result, that result might propagate to the processing of the other network.
43:05
Just to give you an example, this is the Frustrum network, or FrustrumNet Which is a recent approach that really use sequential fusion. In that case, the idea is to use the camera information in order to locate any potential object in the scene. When you have the detection in 2D, you then construct, build a representation of that 2D detection into the point cloud generated by your LiDAR, for example. You know that this 2D object is somewhere within that Frustrum of the point cloud.
43:59
Then you would train another network in order to detect 3D object within that 3D Frustrum. Very, very good results were obtained by using that sequencer approach. Again, you can refer to this paper if you want to learn more about this approach.
44:25
Before we finish our discussion on sensor fusion, I just want to add a few words about the calibration and annotation issue. This is very important, to train the neural network. But in the context of multi-sensor fusion, this become more challenging. The idea is that if you want to calibrate your multi-sensor system, you must have or you must use object that are visible from these multiple sensor at the same time. For example, if you have a camera, LiDAR and RADAR system,
45:12
you could use a calibration object like this one. Which is made of what we call a reflector cone, which is able to strongly reflect the RADAR signal. The LiDAR will also be able to see that object. Because there is a planar form that has been fixed in front of the corner reflector. Obviously, the camera also will be able to see that object. This way, you are able to generate a representation of this calibration object that you move over the scene. And you have that object captured simultaneously
46:04
by all the sensor.
46:10
This way, if in addition you were able to get these sensors synchronized, you will be able to generate a data set. Or that will contain synchronized information from LiDAR, camera, and RADAR. Here is a video where you have, in blue, the result from the LiDAR. In red, the few red point that you see correspond to the point cloud obtained from the RADAR. You have the image information as well. Everything is synchronized and overlaid onto the same presentation. This way, you can use that signal
46:59
that multi-sensor signal, and input it to you your multi-sensor fusion system. Once you have your multi-sensor data, the next step is to be able to build an annotated data set from this multi-sensor data, so you certainly need to have an annotation tool at your disposal. Here is an example of the tool that we developed at Sensor Cortek.
47:36
It consists of a 3D web application that is able to display the multi-sensor data into one frame. As you can see, whenever you have a 3D box and you select that box, you can move it. When you move it, the information will be projected as well, onto the camera image. You really have annotation, in that case, that are simultaneously reprojected onto each sensor. This is quite an extensive task. But that way, this is the only way you can get a very reliable data set, annotated data set.
48:34
There is also another way of obtaining annotated data.
48:41
This can be done using some neural network that you train in order to be able to pre-annotate the information. Let me show you an example of that. In that case, what we did, we use a stereo system of camera. Here you just see one image of that stereo system. We use a network that process that image in order to detect the object of the scene. In that case, the object of the scene being the pedestrian and the cars. The segmentation, we obtain. Once we obtained the segmentation from this stereo,
49:27
we are able to get the range information, the distance to the camera. That way we are able to re-project that 2D and 3D information, 2D plus range information, onto each representation of in that case, the RADAR data. I showed you earlier the different possible representations for RADAR. As you can see, the label, pedestrian, car, are re-projected automatically onto each of these representations. This is a way to completely automate the process of annotation from multi-sensor.
50:12
But in order to do that, you need to have accurate calibration of your sensor. OK. Just to summarize, the research in multi-sensor fusion is still an open subject. It's still very difficult to obtain a good multi-sensor fusion system. One of the challenges, obviously, to be able to capture and annotate data from multi sensor. Remember what we said, why do we want to perform this sensor fusion? Because when the perception of the environment
51:00
is becoming more challenging, when to perceive accurately the environment become more challenging, then you need to rely on the information from multiple sensors. The fact is that the current research done often use data set that are of good quality. By that, I mean data set capture of the fair weather condition during the day. For that reason, you will often see single sensor solution that perform better than a state-of-the-art multi-sensor solution.
51:40
This is just an example that I have extracted from this year conference on computer vision and data recognition. It shows that one method here was able to surpass, and this is a method that used LiDAR only, was able to surpass the result from RJV, from camera plus LiDAR data. Again, this is because basically the data set that is used to measure these performance is of too good quality for multi sensor to really surpass the single sensor. We still need to progress on multi-sensor fusion.
52:34
But multi-sensor fusion is key when the environmental conditions are challenging. And we need to build data set that contain this more difficult driving condition. On that, thank you for your attention.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat