Skip to main content
Loading Explore guides…
Multi-Sensor Fusion · Acedra
Home
Explore
Engineering
Multi-Sensor Fusion
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
101 segments
Copy
Hide video
0:10
- First song is from, again, Ben Eggleton,
0:15
he's the director of the Sydney Nano, and the cloud director for NSW Smart Network and he found The Institute of Photonics of Institute at the University of Sydney. And he's been a very prolific scientist,
0:31
since I knew him from long time ago and welcome Ben. Looking forward to it all. - Thank you, Vincent. Great to be here.
0:40
How does that sound? No? - Do you wanna use that other microphone?
0:47
- Great to be here an intimate audience so we can be relaxed informal, but I know there's also a virtual audience attending.
0:56
What a wonderful conference, fascinating two days really, best conference I've been in person for at least two years.
1:05
But today I'm speaking to you about research in my team at the University of Sydney
1:12
as express Multi-sensor fusion for real-time situational awareness and I'll take you through pretty high level perspective on what we're doing to a non expert audience. And I'll explain initially some of the different roles that appear on this introductory slide.
1:29
So, let's see if this is going to work. So my main role at the University of Sydney is director of Sydney Nano Institute which is one of the university's flagship multidisciplinary institutes we're headquartered in the fabulous Sydney Nanoscience Hub I welcome you to visit when you're on campus. It's relevant to this presentation because it provides breathtaking capability for integrated photonics, and that's all about reduction in size, weight, power of sensors and I'll return to that point
1:58
at the end of my presentation.
2:02
I'm also co-director of the New South Wales Smart Sensing Network which is an exemplar of a translational innovation network sponsored by the New South Wales government. And so we bring together universities in New South Wales to collaborate with end users, stakeholders, and partners on smart sensors that address some of the key challenges in New South Wales, as it relates to environment, transport, medtech, water, and so forth. But what I'm focused today on obviously for this audience
2:35
is the work we're doing sponsored by The World Australian Air Force under the auspices of the Jericho Smart Sensing Lab. And so this is a photo taken back in 2019 at the City Nanoscience Hub when we launched this exciting initiative and in that photo, there is Air Vice Martial Darren Goldie who's actually here with us at the meeting. We've got Paul Scully-Power who also is at the meeting Australia's first astronaut Duncan Iverson, our deputy vice chancellor research
3:06
and other distinguished guests. So what my main business as an academic, as you would guess from that slide is Smart Senses so let me just give you a very high level perspective on smart sensors. Sensors detect changes in energy of a physical phenomenon and converts them into digital data. And so there are various embodiments of sensors that we're all familiar with, that detect aspects of the environment and of course we know that sensors are only powerful
3:38
if we can extract useful information from them.
3:42
And of course we understand that a concept like sensor fusion and AI are really key as we consider sensor fusion and I'll expand on that in the next few slides. And of course, sensors are a vital part of our world as it relates to our health, I think most of us probably we have a Fitbit and we monitor our own health as it relates to autonomous vehicles and as it relates to monitoring utilities and so forth.
4:10
So the concept that I'm gonna focus on today, is sensor fusion. So sensor fusion is a more advanced idea, and it's obviously an important idea it's been around for a long time, but basically the concept of sensor fusion is that you can combine sensory data to create a more accurate, more complete, or more dependable picture of what is happening. And so it's summarized in this lovely graphic where you can see a number of different sensor modalities. So this an optical sensor, an acoustic sensor,
4:43
radar LIDAR temperature
4:46
these sensors then connect onto a platform, a virtual platform or the cloud and there is the concept of the digital cortex, an expression that Greg Cohen who's here referred to a few weeks ago, which is just a lovely way of kind of talking about artificial intelligence that is going to synthesize those different data outputs to provide the situational awareness that is so important here. And of course there are a number of benefits to sensor fusion, that I'll explain and expand on.
5:22
The obvious one is around redundancy,
5:26
multiple sensors to compensate potential down times, uncertainty, production, blind spot coverage, autonomous tasking, and then of course the notion of data fusion itself. So the examples that I'll refer to touch on a number of these different benefits. But I think the concept is relatively killer and relatively well established and it's worth noting, goes back to the early days of robotics because robotics is probably the first example of a sensor fusion technology.
5:55
So of course humans are in fact wonderful examples of sensor fusion the way we synthesize what we see and what we hear and of course we are actually intelligent, so we don't need artificial intelligence because we are actually intelligent. And so this lovely example of, directions passed visual senses to identify sources, and maybe a more subtle example is hearing assisted by seeing, where a visual focus on a target of interest, then underpinned selective hearing
6:30
that allows you to have some noise reduction. Again, very simple example of sensor fusion.
6:38
Of course, probably the industry that's investing the most in our sensor fusion is the autonomous vehicles industry and if you follow Elon Musk he has some pretty strong opinions on sensor fusion technology. So this is a canonical example of an autonomous vehicle that relies on a whole suite of different sensors to give it the autonomous action that it needs to navigate in a world without human intervention. And so, in this example you can see some familiar sensors,
7:13
camera, computer vision, LIDAR, acoustic sensors, but you also see radar
7:20
and of course there is the digital cortex that I alluded to the AI that allows for that sense perception plan and action, which is again at the focus and at the core of the Jericho Smart Sensing Lab narrative.
7:35
The examples that are more relevant to an Air Force Space agenda are along the lines of this slide. So this is a canonical example in the context of, for example, drone tracking, where you might have a layered architecture that relies on in the first instance, acoustic sensors, that provides some very basic location signature that tells you there is an object that needs to be focused on in more detail that might trigger then an optical sensor
8:10
to then try to classify that object. And of course there might be then another layer of sensors that kicks in to provide more complete characterization. And of course, probably the most
8:25
exciting example of sensor fusion
8:29
in the Air Force context is the F35. Now this is taken from a publication. I think everyone's familiar with the narrative of a centi fusion as it relates to F35 but it really is a breathtaking example of a fifth generation fighter that uses very advanced sensors infrared, radar, optical and the fusion of those sensors to provide the pilot that real time situational awareness.
8:55
So a great example. Okay, so this project, as I alluded to three years ago, was established to provide the Air Force with a new way of working focused on smart sensors but as much as it is about sensors, it's actually about how we engage them, particularly in the way we bring design innovation, design picking, our ability to prototype
9:18
new technologies that are very quickly put in the hands of operators and in hands of end users to allow operators to test these prototypes in their environments, often those environments are classified. So these are just some photos taken back in 2009 and again, you've got air vice Marshall, Darren Goldie there looking into the clean room facility which is in the City Nanoscience Hub where we have this fantastic capability for printing microchips and again I'll make a reference to that
9:46
at the end of my presentation.
9:49
So the team is vertically integrated. So we sit in The City Nanoscience Hub and our purpose very much is to accelerate software capability and to be the national leader in smart sensing technology as it relates in particular to providing the Air Force with situational awareness. So on the left hand side, we have that very strong engagement with Air Force operators, fantastic engagement, regular meetings where we are engaging directly with commanders group captains, squadron leaders,
10:22
understanding the end use case, really teasing out
10:27
the perspective around the problem that we're tryna solve to ensure that we're not just solutionizing. On the right hand side, you've got the science, you've got the capabilities that underpin our research and to a large extent we're vertically integrated and that's key to giving us the agility to rapidly prototype and the agility to sometimes say, actually, that's not gonna work and we are gonna pause that, or we're gonna moth all that and we're gonna reconfigure around this other area
10:56
that is obviously a more compelling direction to take. The design innovation sits in there, a vital part of the engagement with rare vital part of our capability to prototype, to understand the use case. Of course, the electronic engineering that allows us to build an optical digital technology that can interface the data fusion, the AI, and then that industrial design again that's around making something that is portable rugged can be field tested.
11:28
And we sit in The City Nanoscience Hub as I mentioned.
11:34
So it is about science and design.
11:38
And that talks to in particular, the construction of a prototype, as I mentioned, but it's more sophisticated than that and that is a journey itself laid out in this slide so our designers are working very closely with the operators very early on in the development of these sensors. We use design concepts, design constructs to for example, sketch out these designs to really meet the requirements of the end user and the human centric approach. The lab testing, of course,
12:10
that's required to validate the performance. We take these sensors out into the field sometimes we get onto a base, sometimes we get ethics clearance to do a field test within our laboratory setting, sometimes we get onto campus and we use our sensors for example, to monitor vehicular traffic. And at the end of the day,
12:32
we're interested in building these prototypes that we hand over to the end users. These prototypes then end up in the hands of Air Force, Navy, Army operators. Sometimes they get taken, as I said into classified environments, and we continue to work with them and understand the feedback to ensure that we can take that to the next level.
12:57
Just two examples that are relevant and fresh.
13:01
The MANTIS prototype, MANTIS stands for Mutual Access Neuromorphic Twin Imaging System. There's gonna be a quiz at the end of the session so you'd better try to remember the acronyms. If you go to the booth downstairs to the exhibit to the corner you'll see the MANTIS prototype that was constructed early 2021 and handed over to Air Force as a rugged portable prototype. It's worth noting that this is a sensor fusion device in that it combines two sensors,
13:31
one of the sensors is the neuromorphic sensor of technology at the Western City university. Greg Cohen, who's in the audience is very famous for being a champion of and developing that fantastic technology and we look forward to working with Greg taking this forward, another example of using the MANTIS on a base. And the second example relates to our pattern of life (indistinct) which was utilizing very low cost sensors that could be placed around a base for example, or a campus in this case
14:02
to monitor vehicular traffic, relies on microphone arrays with edge compute classification, beam forming, and you get a sort of sense of the prototyping and the design thinking that went into constructing these deployable robust waterproof sensors that have geolocate, communications capability, power supplies,
14:24
and this was run on campus and was the basis of a really important demonstration of being able to monitor the patterns and looking for those signatures and anomalies of behavior.
14:36
So great team, just wanna highlight Harjit and Tomonori, who are both in the audience. And again, if you get a chance to go to the exhibit, you can talk to Tom and Haj about the MANTIS prototype
14:49
and a great team that combines science, engineering and design. So of course the context for our research is not autonomous vehicles.
14:58
It's obviously the three-dimensional battle space if you like, and this is a kind of academics interpretation in an unclassified context of what I imagine is the use case in general for Jericho Smart Sensing Lab. So what you've got here is a series of different sensor modalities that are working together to provide sensoring across the different lens scales and timescales. So of course what's different between autonomous vehicles and three dimensional battle space
15:27
is on the one hand, the lens scales that are involved. So the ability to sensor over long distances and short distances and the time scales. And so that motivates a particular portfolio or suite of sensors that we think we are working on and I'll explain and expand on that provide really unique capability for providing that sensing and situational awareness. This is another sort of perspective just to make the point that these different sense of modalities
15:57
have very different performers, attributes, and criteria. So if we think about acoustic microphone arrays I just alluded to fine for on campus short distance robust work at night
16:12
situation, but not particularly good over long distances. If we think of radar, photonic radar, LIDAR, they each have their own pros and cons and it's really only when we combine them as per the sense of fusion narrative that I talked about earlier, that you start to get a more complete robust picture that allows you to exploit the data fusion to provide that more accurate real time situational awareness.
16:42
So a busy slide, but just to get the point across that the team has some really
16:48
unique capabilities anchored in the physical sciences and anchored in particular in photonics and optic electronics and acoustics. And that expertise in the capabilities that I've described allow us to create these sensors and we can prototype these sensors. So just to go through the slide from the top right hand, it will go around from 12 o'clock. Of course, photonic sensors are the base of spaces of spectroscopic sensors that we know we can use
17:15
for chemical analysis and air quality contamination and we've done a lot of work on that particularly through The New South Wales Smart Sensing Network. As we go around, we're working with DSDG on next generation single photon detectors that are the basis of LIDAR, next generation LIDAR for either air or under sea applications. As we go around, we start to see acoustic sensors show up we one scattering is where an optical wave scatter off sound waves and we know that we can build sensors
17:50
that monitor structural health, for example, impact detection and this is work with RF that is unpublished but is an example of a sensor fusion modality.
18:03
Acoustic sensors can be used to detect water pipes. They can listen and hear and they can be given edge compute so they can classify sounds so we can hear all sorts of events as they alluded to on campus. We then see the neuromorphic event based sensors playing the Pfizer wall in that sensor fusion narrative. And then as we go to the top, we start to see that photonics is enabling microwave sensors so whether that relates to RF sensors that can detect microwave signals across
18:38
many tens of gigahertz, or whether it alerts to photonic radar to actually use photonics to enable high resolution radar imaging and I'll end my presentation with that example.
18:51
So just to kind of wrap up then JSSL. So we have very specific capabilities that we bring to bear with our engagement with Air Force. So the photonics, the optical and acoustic sensors that very strong data engineering capability that wraps around all of the sensors and there's key to the digital cortex that I talked about providing
19:13
the ability to classify these objects in real time, the electronic software infrastructure and of course, and their long term and again, I'll come back to this, the ability to miniaturize to integrate the sensors onto a chip thumbnail size using the semi electro fabrication capabilities in the clean rooms. And of course the design innovation wraps around our entire program. The Manchester is probably the best example of a prototype that was constructed in three months
19:41
and delivered to the end user,
19:44
very, very swiftly.
19:46
So just to wrap up,
19:49
it's all about combining the different sensors. Now this is another 45 minutes of conversation to tease out, but just to give you a snapshot of the business that we're in, for example of combining acoustic and optical they alluded to earlier, we might use an array of acoustic sensors to listen, to identify, roughly speaking where a loud object is coming from,
20:15
and it could do that well. It's not particularly good at saying exactly what that object is, but it might really elegantly configure
20:26
an optical system to look. And an optical system might then be able to classify the object very precisely. So it's a nice example, very simple example of sensor fusion, combining a very low cost simple sensor that can be sort of distributed almost everywhere, everywhere, thrown around
20:45
to give you that kind of initial hint of something's there, but it then works with another sensor, it tells another sensor very clearly where to look. We might bring together optical and neuromorphic. So alluded to that earlier
20:59
potential to switch between a visible and a neuromorphic feed because, you know some interesting points for example, a neuromorphic sensor is an event based sensor when there's nothing happening, it sees nothing. And so on the one hand, it's an amazing technology and I'm sure Vladimir is gonna speak and Greg's team can talk for hours about the neuromorphic sensor. But actually you need to combine it with other sensors to really see the picture.
21:23
And of course, thermal infrared, broadband RF sensing and photonic radar.
21:29
So lots of possibilities for fusion and that's the business we're in. Framed around very specific use case directly engaging raft, directly engaging operators, whether it's combining thermal optical and neuromorphic acoustic and neuromorphic broadband, radar and optical.
21:44
Really interesting possibilities, key issue is that we need to consider, so we also need to give consideration to communications between these sensors. I alluded to edge compute goes without saying that in general we like to have edge compute built onto the sensor to do as much as we can on the sensor. Sometimes that isn't always obvious. And that includes the AI as it's scalable and then what about size, weight, power? Some of these sensors start off as a kind of tabletop, laptop size
22:18
how do we scale it down? So these sensors are all kind of smartphone in size. So I wanna end with an example that is actually not sponsored by RF. This is work in my group, published, patented.
22:34
It's of interest to a number of end users and we're working on a potential spinoff company, but it's a nice example of a
22:42
simple sense of modality that uses radar.
22:46
And this was just published earlier this year and got picked up by the online media. There was a lot of excitement around this in some application areas that you might not even have thought through. So the basic concept, of course with radar you all understand the idea of sending out a radio wave at the speed of light and it bounces back of an object and from the time of flight you can detect its location from Doppler shift to its speed and to some extent, the size of the object.
23:15
Now, what's interesting is that as you go to a very high frequencies
23:21
out to say 30 gigahertz, you can increase the bandwidth. And as you increase the bandwidth you increase the resolution of your radar imaging system. So here we are using this radar to image the blades of a drone in real time. So, and this is using photonics. So let me just sort of tease out the difference between photonics and electronics. So this is a more general statement that applies to many of the sensors that we work on.
23:51
As it relates to generating the wave forms that are used in this radar application. So with conventional electronics,
24:00
we do have enormous progress with digital signal processing and ADCs of course ongoing. But in general there is a challenge in terms of bandwidth to size ratio.
24:12
And there is a noise issue. And I guess the real issue that we're trying to tease out with this demonstration is that electronics tends to be hard wired to particular frequencies. Whereas photonics is agile and reconfigurable and I'll show you that. So we have ultra wide bandwidth, we have lower noise, we have multiple functions and purposes and we have the roadmap for integration. So the basic concept here is that we generate the microwave wave form that is utilized to image
24:46
in the optical domain. Which is much, much easier than
24:52
what it would take to generate that wave form using digital electronics. It's very simple undergraduate physics, where we bounce the light waves off a photo detector and we can generate these very, very high bandwidth radio waves. Now the details are probably not particularly important for this audience, but just to give you an idea of what this actually looks like in practice, this is our ability to resolve
25:17
the wings of this drone. And if I knew how to, I could press on that animation and you would see a real time tracking of the blades of this drone spitting around and that works really well. So the last point to make is sometimes you stumble onto applications that you weren't imagining. So it turns out this particular photonic radar that was initially inspired by a use case that relates more generally to autonomous air surveillance, drone tracking for example,
25:48
is pivoting towards vital sign detection. These microwave signals actually bounce off the skin. It turns out not only can we resolve the blades of a drone spinning around, but we can actually resolve someone's breathing and even heartbeat. And so we're working with some partners at the moment on building this remote sensing technology to future applications in vital sign detection. So just an example here of sensors that detect breathing in real time and possibly even heartbeat.
26:22
But I can see that my time's up and my last slide is just to sort of end with that photonic integration roadmap. We're very fortunate to be in the City Nanoscience Hub, as I mentioned. And that means that the sensors that I've described that often start on a bench top the size of this tabletop then are miniaturized into a sort of laptop size prototype that is field deployable in the long term can be printed onto a circuit thumbnail size. And so that reduction in size, weight and power,
26:59
which is genuinely breathtaking, is the key part of our strategy for bringing smart sensors for situational awareness. So on that note, I'll thank you and happy to take some questions.
27:14
(audience clapping)
27:19
- [Woman] Thank you. Thanks for the talk Ben, very interesting. Just had a question about the UAS that you showed. Did it have copper tape or something on the rotors? And what's the real performance for a drone that has plastic rotors?
27:34
What's the performance for a drone that has plastic rotors not metal? - So yes, indeed for the publication, (indistinct chatter) interestingly enough we've moved away from that. I think the general issue of whether you need is a general issue for all microwave systems. Do you need a metallic coating or not?
27:58
It improves it. If you took it away, you probably reduce the sensitivity.
28:06
I don't know the answer to your specific question, we've moved on, so we are looking at vital sign detection here which is interesting, 'cause there it bounces off the skin and you still get a signal.
28:17
I can follow up with you on that.
28:21
What are some of the outputs from the POL sensor? So The Pattern Of Life project was built around the idea that we would put low cost sensors into an environment that would detect
28:36
signatures and that we would monitor patterns and we would then be able to see anomalies. That particular demonstration relied on microphone arrays so acoustic sensors.
28:47
And I think it's fair to say that we understand how that works and these acoustic sensors can have beam forming and classification built onto the edge. So you can actually characterize car, sound, dog, truck,
29:00
and you can geolocate and you can actually then even locate those objects. And of course it can be generalized into old suite of sensors.
29:08
So I'm not sure if that gets to the answer,
29:12
but yeah, so BOL sensors that's the output. Can we get the F35 to self park?
29:21
The next question.
29:24
Don't know if I'm qualified to answer that. - [Man] Any other questions from the audience?
29:30
Well, let's thank Ben. (audience clapping)
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat