Skip to main content
Loading Explore guides…
Karen Liu: Robotics & Graphics · Acedra
Home
Explore
Computer Science
Karen Liu: Robotics & Graphics
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
131 segments
Copy
Hide video
0:06
Thank you everyone for coming. It's an honor to introduce Karen Liu. She's an associate professor at the Computer Science Department at Stanford University. And she has moved a lot throughout the US. What do you just want? Started PhD at the University of Washington, and then she started as faculty at DOC, and then moved to the east to Georgia Tech, and then this summer she moved back to the west to Stanford University almost as if you're still making up your mind.
0:45
So anyway, Karen works at the Intersection of Robotics and Graphics, which is a very interesting intersection of disciplines. I would say, even more especially today because of the relationship with the interest they now seem to real transfer Right. So Karen is very well known for contributions in generating realistic graphics animation, realistic human like motion by embedding physical constraints or forcing graphics to obey physics constraints.
1:21
And sort of that question of generating human like behavior has a natural follow-up which is the same principles that we use to generate human like animation, do they transfer it to generate human like motion in robots right. And I find it sort of very interesting that some of those techniques sort of can be used today, and they are extremely relevant to research today, and I hope that you're going to be able to talk about it. So thank you very much. Thanks.
1:58
Well, thanks for the introduction. Thank you for inviting me over, it's a pleasure to come here to give a talk on how to simulate realistic human emotion. So today I'm going to talk about simulating virtual humans, which is traditionally considered a computer graphics problem for entertainment application. So many of you probably have seen this video from the AGDC this year. Hello. I'm Siren, and I'm a digital human. I was created by an international team of artists
2:30
and engineers who wanted to challenge our ideas of what a synthetic human could be. So creating photorealistic virtual human is not a new thing, computer graphics people had started this 10 years ago. And you probably seen this digital dog giving a Ted Talk earlier this year. But what makes this video or this demo impressive to me is that everything you see is simulated and rendered in real time. And just let me focus on this real time aspect a little bit by showing you an audio clip where
3:06
Siren was being interviewed by a real person in a live show in AGDC.
3:14
You hate the color? I don't hate it, but-- Such a strong word. I think she looks great right? Oh thank you. Looks great? But you can follow it I'm just careful but OK. Want we make it purple? Yes, we can't do that. There you go, it's purple. How did you do that so quickly, live in real-time? Live in real-time, 60 francs per second Oh my goodness. Why don't we actually tell the audience a little bit about copying back to their song live right now at 60 francs per second as you said?
3:53
All right. It looks great, right? But there's something off about this, there's something missing here. And the answer is, it's the motion. So Siren is being driven by a real actress behind the scene being motion capturing in real-time. So all the natural movements and behaviors are coming from a live person. And Siren is really just a digital puppet if you will, although is very sophisticated one. So when I saw this video, I have some mixed feeling as you can imagine, on one hand, I
4:28
was really proud of my computer graphics community and what we can achieve today, but the fact that we still need a live person behind the scenes to provide motion suggests that someone in this community didn't carry the weight and that someone is likely to be me and my colleagues in character animation. But this still highlights just the challenges of creating natural movements and unscripted behaviors, because it really takes a multidisciplinary effort to make any progress here.
5:00
So let's just start out with the sort of going through the process of how we actually mimic the process of a human generating movements today in computer animation. It always start with a thought, with a high level intention like standing over here picking up a coffee cup, and then this thought is going to be turning to some neural excitation, which activate muscles, and when muscle contract it generally torque around joins, together with external forces like gravity
5:32
we will go through a physics simulation process to create a joint configuration at a next time step. All right. So we have a process like that, and if you want to control this then the first question you need to answer is at which level do you control so that your motion will still look natural and human-like but the computation is still tractable? In one extreme one can choose to control the excitation of billions of neurons, but probably no one would do that.
6:01
On the other hand, in the other extreme we can completely bypass physics and directly control the joint configuration at the end, but this kind of a kinematic control usually doesn't work very well if your environment is dynamics. So that's our first question to answer. Where do we control? And the second question is how do we represent this thought computationally? We could literally represent as the fourth stayed at every millisecond by following
6:31
some kind of trajectory, or we can represent this thought as just an abstract goal without detailing how we going to achieve that. OK. So let's examine this space a little bit. The hardest part to achieve is this corner, upper right corner, state of the art techniques still can control at the level of muscle activation to create complex motion without reference trajectory. On the other hand, most of the application you've seen in entertainment industry can all be placed in this corner.
7:05
In this corner, motions are generated by design by artist or directly captured from the real world, it can give you a really high quality motion, very natural looking like the Siren demo I showed earlier. But this is at the cost of a generality because these characters or these agents don't really respond to physics, changes in the environment, and they can't really do anything that's beyond why you're describing the trajectory. And the research that I'm mostly interested in,
7:33
the region that I'm mostly interested in my lab is this area. So what's special about this region? First we don't depend on trajectories, the agents need to be able to plan its own movements based on abstract goal. And second we control the joint work level, which means the motion has to come through the physics constraint, it has to use physics to simulate motion. So with that we're able to create the virtual agent that can respond to unplanned physical probation
8:03
in the environment, we can also train an agent that can transfer a skill to a different environment, in this case, we use reinforced modeling to train the agent to ride a bicycle on a flatter train and then we just transfer that to a very different situation. Our agent is also autonomous, meaning that it has to solve an optimization problem to plan its movements according to some abstract goal. In this example we placed the agent in the mid air and started with some initial random velocity.
8:35
An agent has to come up with a plan on its own to manipulate the body so he can land safely on the ground, in this case the goal is really abstract, it really just say don't die, and then the agent had to figure out. In a different example, so another thing that I want to focus is sometimes reference trajectory is not easy to get. So if we don't depend on trajectories it has really give us a freedom to generate motion where the trajectories are hard to come by,
9:05
like I still haven't figured out how to do this in my motion capture lab, but so we end up learning from scratch using reinforcement learning. And in a different example we try to teach agent to be modest, to put some clothes on themselves, and turn out this reference trajectory is also hard to motion capture because because occlusion of clothes on the human body, and come to think of that that's the whole point of addressing, right? You are including your body using cloth.
9:39
So if we don't depend on reference trajectory that means that the naturalness of emotion has to emerge entirely from the task and in the physics which is a pretty difficult, while there are a lot of existing methods that demonstrate high quality motions by imitation learning or mimicking trajectory we really believe that the that he did that the natural this should be an emergent property in animal movements. So we propose to improve, to achieve the naturalness
10:07
by improving the anatomical realism of the model instead of just reducing to tracking a trajectory. Simply put, instead of going left we want to go up. To be more specific, most of the virtual agents in computer animations are actuated by independent rotary motors, very much like a robot. To directly control joint angles, joint torque is nice because you make simulation and control and modeling easier. But the motion we generated are often being criticized as being robot like which shouldn't
10:48
be a surprise to anyone. So we hypothesized that the naturalness of emotion will emerge if we use muscle activation as actuator. And this hypothesis has been tested or had been supported by previous work. People have tried to simplify muscle model on a relatively complex task, or a forward muscle model on a simple task. But creating or using forward muscle model to create complex motion without reference trajectory is still very hard today. And this, there are many reasons this
11:23
is a really difficult problem. For one, our muscular system is highly redundant. We have more than 90, our legs, we have a more than 90 muscles to control just six degrees of freedom. And every single one of those muscles will add some complexity to the dynamic system. So in a recent work, we are asking the following question. Is there a task-agnostic approach to generate natural motion with comparable to forward muscle model but only using joint work?
11:54
Because it's simpler to model and faster to compute. And we want to emphasize the approach that we're looking for is task-agnostic. We want the space of actuation defined by anatomical model rather than what task you are performing. So in order to answer that questions, the first thing we want to do is just to formulate a very generic optimal control problem in the muscle space in the joint torque space and compare the differences between them and see if we can get some insight from that.
12:29
So on the left, we're solving a colocation problem to find optimal muscle activation trajectory a. And the joint angle or joint configuration trajectory q. So q is the final motion that you will see on the screen. And we want to minimize the muscle usage and the penalty, a task-related or task-relevant penalty term. This term could be very, very generic. It could be you know how far you are away from the final joint configuration you need
12:59
to be subject to articulate rigid body dynamics, the muscle dynamics, and the bounds of the muscle activation, and the bounds of a joint. So the last two inequality constraints are basically just the balance for the control variables in the state variables. On the right-hand side, we want to solve exactly the same problem but with a different control variables. This time our control variable is a joint torque tau. And here we're trying to show, to minimize
13:29
the usage of a torque in the same task-relevant penalty term subject to articulate rigid body dynamics, the bounds for the torques, and the bounds for the joint, basically, joint limits. So if I solved these two problems separately, will I get the same q? No. Of course not. Right? They're different. They're different constraints. And there's no reason to believe they will be the same. But the question I'm more interested to answer is, is there a way I can transform
13:58
the problem on the left, the muscle problem, into the joint torque problem, an equivalent problem? So I can solve in the joint torque space, which is the easier to model, simulate, and control but still enjoy the benefit of the muscle. Of course, I would want the solution I get will have the same minimum as the original problem before I transform that. So to answer that question, first let's just cross off all the terms that are identical between these two problems.
14:28
First, the task term is the same, the articulated rigid body dynamics is the same, and the joint limits are the same. OK. So what's remained is the differences between these two problems. And we identify, the first difference is the torque bounds, the joint torque bounds. Because the on the right, it this joint torque formulation which doesn't take into account the muscles at all. So you can imagine you can easily create some kind of torque patterns
14:53
that are not possible to be achieved by muscles, lies between 0 and 1. So that's the first difference. And the second difference is the energy assignment. You know, we're using this torque energy, sum of torque squared, sum of square of the torque, which doesn't take into account the muscles. And you know, your energy assignment will be, of course, different. So let's first focus on the joint torque limits difference. If I were defending myself against someone bigger than me
15:23
in a self-defense class, the first thing they would teach you is to use this thing called arm lock technique. So basically you try to figure out a way to put your opponent's arm behind his back. And anyone in that position will not be able to generate a lot of torque. OK? So the first thing we learned, the first observation is that joint torque limits are state-dependent. And other things about our muscular system is that it's very complex. We all know that. But the muscles, some other muscles
15:51
can span multiple joints. And some joints have multiple muscles around them to generate torque. So the second observation is that two joint torque limits also depends on other torques. What other, how much you are activating other part of your body would affect the torque limit at this current moment. So to be more-- to make it more concrete, let's go back to the torque bounds that we usually use. So this is like a box constraint. Right? We have a upper bounds and lower bounds.
16:19
And there are just some constant values. So your constraint looks like a box. By observation, while we know that it should be at least that-- the two joint problems should look at least like this. It's a function. The bounds should be a function of your state. But that's still not enough. By observation too, we know that it also depends on torque. So essentially, eventually, we use a-- we define a implicit equation to define the range of human joint torque limits.
16:49
We hope this function C can capture some of the muscle capability. But only use the information available in the joint torque formulation. So the next question is how do we define the C? And how do we evaluate that? We define C being minus 1 if there exists a muscle activation such that the desired torque can be generated using valid muscles at a given state q and d q. This is a famous forward muscle dynamics which maps a muscle activation to its corresponding torque condition on the state. OK.
17:28
So how do we evaluate this? The short answer to that is we use a software called OpenSim. It's a biomechanically realistic simulation tools that allows you to compute this function, for dynamic function. It's a software widely used in biomechanics and system development at Stanford. But still, if you look at this, in order to evaluate this function, you need to solve a feasibility problem. You can relax that problem through an optimization problem. But still it's a very involved process.
18:00
So we decided to approximate this process of solving this optimization problem using a neural network. How do we train this neural network? That's a more interesting question. Right? So who can provide the training data? We utilized OpenSim, a muscle-based simulator and modeling tool to solve one millions of those optimizations problem for us. And we used a very detailed model to do that. So basically, we can train a classifier that's learning from a very detailed muscle model to do that.
18:35
It's also task-agnostic because when we sampled the space, we uniformly sampled in this R15 space. So with that, we can replace the simple box-like bound with a learned joint torque limit C which is task-agnostic and state-dependent. In fact, that this is represented by. neural network is great. Because you know, when you solve for your optimization problem or optimal control problem, you know that you have to evaluate the function. But you also have to calculate the gradient of that.
19:06
And then they can be both done in a very efficient manner. Let's take a look at some comparison. So here we want this agent to jump as high as possible. The green agent is constrained by the conventional joint torque limits, the box constraints. And the blue agent is constrained by the learned joint torque limits.
19:33
OK. So first thing, you notice that, oh, the blue agent can jump higher. It must be better, right? Because that's what you want to optimize. But if you look at closely, you notice that the reason the blue agent can jump really high is because it hyper-flexed its ankles. Why would it do that? Because when you hyper-flex your ankle, you can get lower, which would give you a longer distance to accelerate before you leave the ground. So the optimizer figured this out. Except for, nobody can do this.
20:03
No real human can do this. When you're in that position, your ankle's in that position, you cannot generate torque. You cannot accelerate your center of mass that fast. And our blue agent figured that out, the constraints, without using muscles.
20:20
So we compared this with the muscle model. If this is like, our ground truth, our baseline. We solved this through OpenSim. And you know, what we would like to see is that we can get a similar motion without using muscle but with a better computation time. And in this case, we have a little bit of save. Because it's very simple problem. But we can still save half of the computation time. Later on, you'll see that the save was a lot more significant when we move to RL, reinforcement learning.
20:52
And since our learned function C is task-agnostic, we should be able to just apply this to a completely different task, and which we did. So in this case, the agent wants to accelerate as much as he can before it release the bar. So they can fly further. And now you can see that the green agent is doing the same thing. It's overbent its knees and hips so that he has a longer distance to generate torque. But again, you know, no humans can do that. When you're bent, bend your knees that much,
21:22
you cannot kick that quickly. Here is a comparison with the muscle model. And we see similar behaviors and with a better computation time. So that's-- question?
21:41
Exactly this, are slightly different. Yes. So we solved it. So this is not RL. We're not solving a policy. We're solving a trajectory. And we formulate it as a non-linear optimization problem. We're solving probably s, q, p. I need to look it up. But yeah. I need to look--
22:06
Yeah. So this problem is simple enough. You know, and a local minima is actually, kind of, what we want. Well, it's not like we tuned the initialization a lot. Initialization is really just the person stay there over the bar. But you know, in general, local minima will play a role in this. But for a simple problem like that, where not suffering that local minima issue. OK. So that's the first part I want to mention. And then there's the second part. Right? How about the energy assignment?
22:42
That's another reason that you would end up getting different trajectories. Because you assign energy differently. So I'm going to talk about this power really quickly because there are a lot of things I want to get to later. Just give you some motivation motivating example, if I try to pull myself up from a bar and with my palms facing outward, that's called pull up. If I tried to do that with my palms facing inward, that's called chin up. Which one takes more effort?
23:10
I didn't know because I couldn't do either. But my student told me one of them. Which one, pull up, or chin u? Pull up. Pull up is a lot harder than chin up. So we know that the muscle-based effort is also state-dependent depending on, you know, where you stay is. But if you look at our current energy function, most people use, they can't tell the difference. If you do that for chin up, and calculate the torque usage for chin up and pull up, you get exactly same number.
23:38
So we need a better energy function which is state-dependent. And I'm going to skip all the details how we formulate that energy function and just jump to the punch line. Adding in, we trained a neural network to predict energy based on the state and the torque. And how do we train that energy, or train that neural network with our regressor? We used OpenSim again. We learned this from a model-based about mechanically realistic model and train the energy assignment from that. OK?
24:16
And with that, you know, we can prove, we actually proved in the paper that we can transform any muscle problem. Formulate it in this way, to an equivalent joint torque problem and reach a q with, q is the motion with the same minimal value comparing to the original problem. And now let me show you a little comparison of using the energy function we learned versus the simple sum of torque square energy function. So the differences are more subtle here because this is the energy function.
24:56
But you can still see the difference here. The blue agent has more passive movements in its knees. And this seems to confirm the biomechanical hypothesis where animal movements-- animals tend to use a lot of the elastic elements in their body to save energy when generating torques. All right. One premise that I alluded to at the very beginning is the RL. And Russ?
25:30
Yes. Yes. Correct.
25:39
That's exactly correct. So I didn't cover the details of that. Co-contraction is something we cannot do. And while we don't have here, or the information we're missing here is the length of muscle. Right? That also has its own, you know, ODE to govern that. And this is basically well what we call internal muscle state. And that's something that we cannot capture right now with this model. And that's definitely something worth thinking about for the future work. Thank you. All right.
26:10
For RL, how do we apply the models we learned to RL?
26:25
[INAUDIBLE] Yes. Yes. Yes.
26:32
No. No. So C and E don't depend on task, don't depend on the environment. But when you have a different task, you have to formulate your optimization problem differently. Right? The task term will be different. And the contacts, depending on what kind of contacts, you probably have to do that. So your optimization problem will be different, but the C and E will be the same. Does that make sense? [INAUDIBLE] OK. Yeah. But yeah. Like, for example, if you want to simulate something that's
27:08
not holding versus not holding, you will have different contact constraints. Right? But it's not really-- Yeah. OK. We can talk about that. So RL. There's been amazing progress in deep reinforcement learning in the past few years. With a motion capture reference trajectory, people have done a really good job learning very complex policies. They can even do that with a muscle model, that forward muscle model, if you have a reference trajectory. In the case of a cell reference trajectory,
27:41
you cannot do imitation learning or tracking. Then it's a lot harder. We're still able to learn some reasonable local motion from scratch through curriculum learning. But it's a lot harder. And for muscle model, it's more directly learned in the muscle space. The best we have seen is in 2D. OK? So how can we use this learning, this joint torque limit C and the energy function E, to help RL? For energy function, that's easy. You just stick it in your reward function
28:13
and replace your regularization term with this learned energy function term. For C, the torque limit, joint torque limits, we use it as an inequality constraint in the physics simulation. So when your policy propose a tau, we'll first check if it satisfied this inequality at all or not. And if it doesn't, then we'll project the tau proposed by a policy to the boundary of this implicit function, which also make inputs as a function a good choice. Right? Because it's a better representation
28:46
for the function. So essentially, we redefine the action space to be more anatomically correct for generating human motion. And with that, we were able to learn a few locomotion policies. I want you just to ignore the upper bodies, upper body motion was horrible. And can you just ignore that? The reason why I want you to ignore it is because when we learn C and E, we only learn from a lower body muscle model. And also, we don't have any terms
29:17
to regulate upper body movements in our reward function. So if we don't do anything and just learn like the way we train a robot, this is kind of walking you can get. It's not bad. It's walking. Right? But if you just change your reward function by using E and change your action space by using C, you can get much better move movements. And if you directly learn from muscle model, we're not able to find any policy that can generate functional gait. And this is for running All right.
29:57
So one thing I have not talked about is this joint limits. And I'm going to just talk about that real quickly. Your internal, this box-like joint limit is also really bad. What I'm showing here is the range of a motion of a human shoulder. It doesn't look like a box. So when we know that, you know, you range of a motion is definitely not like, your human range of motion is definitely not like a box. It's not something that box constraint can describe. A second thing about our joint limits
30:32
is that it's also state-dependent. If you try to bend your elbows when your arms in front of your torso, you can bend almost 180 degree. But if you move your arm behind your back, the range of the elbow is highly restricted. So again, going back to this joint limits, what can we do with this, this box constraint? First thing we know is it has to be state-dependent. But again, it's a much more elegant way to express this is to use the implicit function like this. How do we learn this function D?
31:05
Well, luckily this time, we have real-world data. Michael Black's lab at MPI has this really nice database with a extensive poses captured from real humans, human athletes. So We learn our function D using this data set. Once you learn function D, how do you enforce that? For this, we have to enforce that in a physics simulation. You need to be able to use constraint force to calculate function force such that the next time step, your q still satisfy the E poly constraints
31:40
by formulating this linear complementarity program. More important thing here is this D, being a neural network, make computing Jacobians very, very easy and efficient. We're just going to show some results here. This is a complete passive agent. It doesn't have any internal forces. It doesn't generate any internal forces. It just fall on the ground under gravity. So on the right, this is how people usually generate ragdoll. Right? They have this box constraints like I talked of before.
32:11
It doesn't look that bad. It's just a ragdoll, you know. You probably wouldn't notice a difference. But if you use a learned joint limits, the motion looks a lot more human-like. And this is without any control. It's just the fact that you set the joint limits that's a little bit more human-like, you can get something already better without control. And now if you have this function D, it can be also really useful for solving inverse kinematic problem. Right?
32:37
Basically, your solution will be constrained in a set of a state, set of poses that's human possible.
32:46
So this is what we usually can get. And when you get something like that, you have, probably wouldn't even think that's something's wrong with that. But if you really think about what humans can achieve, then, you know, our model can't correctly fail to reach the target because of the human limits. And what we didn't expect in the beginning was this, that the benefit of D beyond-- just you know define a better range of human of motion-- it also give us self-collision avoidance.
33:13
When you do IK, you probably get this a lot. Unless you have a really nice self-collision routine, collision checking routine, it could be expensive. Sometimes you want to do in the real time. But if you use D as a part of your constraints in solving IK, you get the self-collision avoidance for free. And the reason is because when we learn this from the real world, no real-world poses will go through its torso. So you just learn that automatically.
33:43
Now with a better replacement, by replacing this with a better learned joint limits, we complete our framework for biomechanic, more biomechanic framework for simulating and controlling human motion. To summarize that, the ingredients is to identify the three functions in a generic optimal control problem, replace them with free neural networks, learn from the real-world data, or find a detailed biomechanically realistic model. All right. So this work was done by-- all these works
34:17
so far is done by my student Yifeng. He was a master's students at Georgia Tech working with me and now he's a PhD student at Stanford. So now we know a little bit about how to synthesize. So simulating realistic human motion. How can we do, how can we use virtual humans for robotic applications? I can think of at least two places a virtual human could be useful. We can teach robots to move like humans. Or we can use virtual humans to teach robot how to interact with human.
34:50
Let's start with the first part. Ideally, I would just like to build a virtual robotic simulation and train a policy and then just transfer it to their real world. Right? That's the ideal situation. But apparently, this doesn't work for a task that's complex. And so this means that if I want to ever want to teach a robot to move like humans, I have to solve this problem we call sim-too-real, or reality gap. Basically, when you train a policy in the simulation,
35:22
how do you deploy to the real world? And researchers have hypothesized a very long list of risk factors responsible for sim-to-real problems. Today, I'm only going to focus on this dynamic model discrepancy. That's one aspect of a reality gap. Model parameters is a list of the numbers that define the dynamic models and the environment. Let's start with something simple. Suppose this is a space of all model parameters. Every single point here is a dynamic model.
35:56
Given the robots and the simulators, I can train a optimal policy for that point, for that dynamic model combination. But it's like, the real world dynamic is likely to be different. So if I transfer the policy I learned for the green star to the red star, it's going to fail very quickly. In general, there are two ways you can think about to mitigate this issue. One is to train a robust policy that can handle a reasonable amount of noise and then transfer to the real world and hope it will work.
36:29
This works in some cases but usually, you know, your real world has to be somewhat, your simulation has to be somewhat close to your real-world dynamics already. Alternatively, we could train a dynamic model using real-world data, and essentially, want to bring the simulation closer to the real world. The limiting factors here is the amount of real-world trials could be large. And it's usually just more time consuming and more effort to acquire those samples. So we propose a different approach
37:04
to tackle the issue of a dynamic model discrepancy. Suppose you have infinite amount of compute such that you can train the policy for every single point in the space. Suppose further that you have a very fast algorithm that can tell you which dynamic model matches the current observation of the motion. With these two scenarios, during real-time execution, all I need is every single time step, I look at the current history or short history of the robot
37:35
motion, determine, select which dynamic model, and then apply the corresponding optimal policy. But in reality of course, you can't do this. You cannot possibly explore the whole space and prelearn all the policies. But with the advancement in deep learning, deep reinforcement learning and physical simulation, it might be possible to train, to explore a continuous subspace that matters to your task. So with that in mind, we propose a universal policy which
38:05
is basically, it's a policy trained under reinforcement learning framework. But it's different in that we also take model parameters as a part of an input. So we call this a universal policy because the action is conditional on the dynamic model. Now the question is, who is going to give you that value. Right? We trained another neural network that predicts these values based on the short history of a state-in-action pairs. We call this online system identification.
38:36
Once they are trained, we should be able to apply this to a new environment that you've never seen before. And it, kind of, worked. So we tested as on the simple hopper, training a hopper, that hops on a surface, it's unknown to it. So basically the hopper doesn't know where the friction coefficient is. What's the friction coefficient at different sections. And the hopper doesn't have any vision. So with that, the hopper, so hopper really has to use its history of its motion
39:09
to identify the model parameters in real time. But we quickly realized that this only works in the very simple case. The biggest problem here is this online system identification has to perform very well in order for this universal policy to be effective. So if you give this our license identification with it as history that its never seen before, it's likely to make a mistake. When it makes mistake, the universal policy would just be confused. You would be very suboptimal.
39:42
So essentially, this is a zero-shot transfer approach. Maybe we're too ambitious to think anything we train in simulation can be just directly applied in the real world without modifying, using any real-world data. So this issue could be mitigated by a very simple idea. Most of the engineers start off their process with a system ID. They identify a system and then they learn a policy. And system identification done this way is usually trying to find mu, to find a model parameter
40:17
that best match the observations. And the observations are usually generated by some random policy. With a universal policy, we essentially propose reverse order. Let's learn the policy first and delay the system identification until the moment we want to deploy it to the real world or to a different environment. If we can learn as, we can do system identification after the policy is available, then we can directly find a mu that maximizes the performance of a policy for the task
40:50
it would really care about. OK. So the difference here, the biggest difference here is that we, at the end of the day, this mu may not be the true mu. But we don't really care. We just want a policy to be able to perform well and forgo the mission of looking for the true model parameter which might or might not matter to the task. All right. So a quick illustration using my little diagram here. So remember this black region is where the universal policy is trained for.
41:23
And the red star is the true reward dynamics. If you're trying to match, find the true star by matching the observation, it's very difficult. But if your objective now is just to find the mu that makes the policy work well, then it becomes easier in many cases. Because there might exist multiple mus. And your goal is to just find any of them and forgo this quest of looking for the true red star. Any of them will get your policy to do what the task you care about.
41:56
So with that idea, we modify our online system ID identification to post-learning system identification. And during training, we just train universal policy under a distribution of mu. And during testing, we use real-world data to directly optimize mu for the task. OK so with that, we're able the transfer a little bit further. We can transfer a policy, transfer on one simulator to a different simulator. You might think this is a nothing. But there are a lot of differences between simulators.
42:33
You know, because the DART is one physics engine that we create in our lab, so we know the difference pretty well. Here's another example.
42:45
All right. There's still a problem here. And the biggest problem here is this mu cannot be too high dimensional. Remember we're going to use the real-world data to optimize the mu stochastically. And if the mu has more than five dimensional space, it's going to be really hard to do. So instead, we will train a projected network together with a universal policy that maps the model parameters to a low dimensional space we call a compact space c. And we call this projected universal policy.
43:21
And during testing, we will directly optimize this c. Find the optimal c that make a projected universal policy work the best. And this time after we've done that, we're able to finally transfer to a real-world hardware. This is a Darwin OP2. I don't know if you have worked with that. It's not easy to work with. But it's, you know, we can afford if. It's Less than $10,000. But it could be a very good platform for sim-to-real because very challenging.
43:52
It doesn't have a very good sensing and actuating capability. So nevertheless, we're still able to get Darwin to do something interesting. So this is the first iteration after universal policy has been trained in simulation. But before it's adapted using the real-world data. So what you're seeing here is the reality gap. Right? In the simulation I'm going to show here, it walks fabulously going forward. But it doesn't work in the real world. But after 20 iterations, we can see that Darwin
44:23
start to move forward. The universal policy being fine tuned better now. And by iteration, I really mean just like one sequence when walking. So you let a robot walk 20 times. And they can do it, and which takes around 15 to 20 minutes of training. We're also able to get Darwin to walk backward, almost. And sideway.
44:56
Yes. You can do it. Yeah. So that's definitely some promising result. But we're still not happy with the robustness of our policy. We're also thinking of some transfer, some bigger reality gap. We want to transfer to something that looks even more different. One thing we noticed, that the testing and training process are very different in our current framework. During training, we're training this projected network. And there's a low dimensional space, the compact space.
45:26
But this mapping was trained without using any real-world data during testing. So this begs the question, will meta learning be useful? Can we expose our learning agent to the same process of training and learning, sorry, training and testing, such that this mapping can be suitable, can learn something suitable for fast adaptation to the new environment. So with that idea, we come up with a very simple algorithm. It's very similar to the meta learning.
45:58
Here instead of-- we simulate or we sample the distribution of a model space. But you can think about this as like a task space in meta learning analogy. So every iteration we sample, this is the space of the model parameters. For every single sample, we fix the current algorithm, sorry, current policy. And look forward to optimal c. And we do that for every single sample. And once we have the K pairs of mus and c's we use those pairs to update the gradients of the policy.
46:33
And then we'll repeat the process. So now with that, our algorithm, our training process looks like this. We are optimizing policy under the optimal c for a distribution of the environment and testing, sounds like this, we are executing the policy under the optimal c for the real-work environment. So they match much better now. We're training what we're doing the task for. And that's, sort of, the idea of a meta learning. And with that, we're able to train a [INAUDIBLE] to move after--
47:13
we're just going to show that-- 13 iterations.
47:20
And we can also change the environment even more by weakening one of the motors of the robot to create a three-legged dog. And the robot is still able to perform the task successfully. This is it going uphill.
47:41
All right. So this work consists of four papers all from my student, Wenhao. He is a fifth-year PhD student. And he's going to be on job market this spring. All right. So I'm also interested in using virtual human to teach robot how to interact with humans. Assistive robot has the potential to provide more intelligence, physical assistance, to people who need that kind of assistance. And usually, designing a controller is difficult because you have humans in the loop. And you also have to--
48:18
and the humans that are moving oftentimes in a lot of the interaction. And so physical contacts. So my lab has been working on this problem we call assistive robot, robot-assisted dressing which is basically we want to exactly this. We want to teach robot how to put clothes on human. This is a multifaceted problem. Because you have a joint optimization between humans and robots through an [INAUDIBLE] dynamic system, which is the cloth and that's not. It's highly nonlinear and deformable.
48:51
And you might think a cloth is harmless. How can a piece of a garment hurt you? But a bad robot can still use a garment to hurt you in a very unpredictable way, let's say. So this is a video of doing training. This is an early stage policy from reinforcement learning. And I think is it fair to say that you know this nobody would want to put someone you know going through this. So there is definitely a value of using virtual humans to do learning, especially when you use the learning
49:26
approach to train robots. And another benefit of the virtual humans is that you can model, we can modulate the kinematic and dynamic capability of a human. In this case, we're training, trying to train one robot policy to interact with a wide range of human behaviors. What you're seeing is the same robot policy. But one is interacting with a weaker person. It has to modify the action. Here is a slow motion. You can see that at the end, the robots end up
49:56
doing most of the work for a person with a weaker arm. Similarly, we can modulate this condition called dyskinesia. Basically, it's a kind of a tremor or people with a limited range of motion.
50:14
So besides using robot or using virtual human in place of a real human for training robots, we can also use virtual humans to collect data for us. You know, vision perception is useful in many cases. But for dressing, it's limited due to the occlusion. But we can largely enhance robot's haptic perception. Essentially, we want to teach robot how human feels when the robot applies certain action to humans. Because robots don't come with this kind of common sense
50:43
that experienced human caregivers have. So specifically in this project, we are trying to teach robot or predict the or estimate the pressure applied on the human limbs from the force and torque sense as robot's end effectors. So we used the simulated results, class simulation, to generate a lot of training data. And then we trained a neural network to do this prediction. And here's the quick result. On the left, this is the estimated result. This is the pressure map that robots
51:16
think is happening on human's [INAUDIBLE] based on past three vectors that it sends from the end effector. And on the right, this is the ground truth. We call it ground truth. But this is simulated result. Later on, we have a follow-on work to compare with a real pressure data from, measure from the real world.
51:40
We also extended our simulation framework to other activities of daily living that you might want robot's assistance. For example, drinking, bed bathing, feeding, as well as just general manipulation of a human limb.
52:05
One last thing I want to mention is we can also use a virtual human to train exoskeleton. There exists a lot of assistive walking devices which ease the burden of a local motion for people with motor impairments or older adults. And these devices have a sensing capability. They can sense the gait. It can also provide activation around the hips. So our idea is to augment this device for fall prevention. So what we would like to do is to use on-board sensors
52:36
to predict the probability of falling at every single time step, a single moment. If a fall is detected, we would like to use on-board actuators to modify the gait, maybe make the gait a little bit wider further, or make the person take a gait a little bit quicker. So the first thing we need to do is to view the virtual humans that can walk with this device. So this is a walking policy wearing the device. And this walking policy is really important. Because later on, we're going to use
53:11
that to train the exoskeleton. So we did a lot of extensive comparison with the biomechanical data. And once we have a confidence that this policy is similar to human's behavior, we can first use that to collect a lot of data for training this fall detectors. And we don't have a problem of gathering a lot of negative examples because this is all in virtual. We can push human as much as we want and train the fall detector with a high accuracy.
53:40
And after we have a fall detector, we could use this virtual human again to train recovery policy on the exoskeleton. And here is the comparison. It's all in simulation. This is really early on work. We have not applied this to the real exoskeleton yet. That's the next step. All right. So I want to end my talk, it's about time, with this video I started, the siren demo. I know it's very controversial, but I'm going to say it anyway. I think a computer graphic is quite
54:12
ready in creating virtual life. But we still have a long way to go in terms of generating realistic and naturally intelligent human behavior. And if we really want to do, you know, get rid of this Wizard of Oz behind the curtain, we have to rethink what virtual humans mean to us. It's not just a digital puppet anymore. It should be a autonomous agent that can perceive the world on its own. With that, I'd like to take my funding agencies. And thank you for listening. [APPLAUSE] All right.
54:50
Any questions?
55:01
I'm just curious. For the exoskeleton that you are modelling and testing, what are its capabilities. Like, is it able to help an individual who-- To walk. Yeah. Can't walk at all? To walk? Is it only a little bit of assisting at the hip? Right. Yes. So that exoskeleton is a real thing. We modelled if off of a real exoskeleton designed by Steve Collins, which I heard that he was just here a few days ago. And we're working together now. So that capability-- so the design
55:31
was for walking assistance, not for falling prevention. So they have a very-- the exoskeleton itself can generate 200-- I think, don't quote me on the numbers, that's what Steve told me-- of Newton-meters. But you wouldn't want to do that on the real humans. So usually we cap it around 25 to 30 Newton-meters. So that's one constraint. Another constraint is that you cannot charge torque too quickly. So the frequency of a control is also limited at 15 Hertz per second.
56:08
So those are sort of the constraint we had to work with when we developed policy.
56:19
I'll ask a question and then-- so I was listening to you while you were explaining the first part of the talk, how you were, sort of, replacing each one of the constraints by a neural network that has been populated with data examples. And at the beginning, I started-- OK, so we replaced that constraint. But there's still a role for, sort of, preserving the structure. Right? I might still want to enforce that. I have torque limits. And then you, sort of, replaced that one also
56:54
by another neural network. And then the third constraint by another neural network, which brings up the question of, why replacing them one by one and not just like, it's one full, sort of, constraint that maps everything. Yeah. Yeah. We actually tried that. And then as a, sort of, baseline comparison. So the first thing-- most people would just say, why don't you just train a function that maps the torque you used to muscles? And then I can just use that to simulate.
57:28
Anyway, I can still solve my control problem in the torque space. First of all, this function is one-to-many mapping. So you can't really train a function to do that. You can make assumptions. So when the torque maps to infinite number of activation, I only want the one that has as small as L2 norm. You can make those assumptions. And then you can do something like that. So that's actually what we did when we wanted to create this baseline. But at the end, it's still, I guess doing something
58:01
like that, it becomes more task-dependent. Because you can't really train with high accuracy when your output is high dimensional space. So the training is harder. And you're just training bigger space. And you might argue that just have more data. And I think that's probably true. You can solve that problem. But at the end of the day, the real issue we don't like is that you still have to keep two models around. You have to have your skeleton, like,
58:28
the articulated rigid body and the muscle model for your simulation and downstream tasks. So what we really want is to just do everything in the articulated rigid body space with a joint work activation and not worry about muscle at all. That hopefully answers some of your question. We have a question over there.
58:55
Thank you for a great talk. So actually, kind of, connected to the previous question. Do you feel that this approach could naturally connect to understanding better model dynamics of objects like you say, neural [INAUDIBLE] or something like that. Or is there a fundamental difference that doesn't automatically extend another environment? So let me see if I got your questions. You're asking if this can be connected to objects, to understanding-- sorry, I don't think-- [INTERPOSING VOICES]
59:23
--behavior is hard to understand. Let's say things are very [INAUDIBLE] Oh yeah. I Yeah. I think so. Let me just complete the question you're asking a little bit. So let's just say, if I do the same thing with objects, can I do this end-to-end learning versus something that I, sort of, replace component by component? Just because you said that is related to the previous question?
59:50
I actually tried something similar to that too. Like, you could just train, you know, a neural network to predict directly next state. A lot of computer vision people have done that. And I think if your downstream task is really just know, just going to fall down or not, maybe you can do that. But if it does-- what you really care about is that state, whether that state you generated is actual or not. I think it would be better off. You still keep this model, the physical model,
1:00:20
as intact as possible. And then just replace the part where you think that data could be helpful. Give you a quick example, a very simple example. When we trained this compact model, when we tried to improve our rigid body simulator, we only improved the part where that represent the friction [INAUDIBLE]. Because we know that the [INAUDIBLE] friction is the empirical model. Right? So maybe by just learning that part and then keeping everything else,
1:00:48
like the Newton dynamics still intact, is the best solution. And we also showed that it is indeed the most generic way to predict future.
1:01:03
Justin.
1:01:07
That's a really great talk. I was in computer graphics applications. We don't really care about whether everything is perfectly accurate physically. Right. And maybe what matters more is plausibility is some cases. Like, some of the motions that we saw, it just looked awkward. But of course, physically, they maybe better to realize. And in some animated environments, you know, these characters do totally non-physical and weird stuff. But when we watch the films, we're
1:01:33
really perfectly happy to accept it. Is there any hope of like, in that, sort of, an application incorporating some term that measures like, plausibility. Or I don't know the right term. Yeah. All right. So I think it's hard to answer that question really. But I think when you get your model more accurate, the functionality or physic plausibility and possibility become closer. I feel like that's probably the best I can answer that question. Like when something, when your model becomes--
1:02:08
you know, just look at the blue agent versus the green agent. For blue agent, it's impossible. But that also is aligned with possibility.
1:02:26
I'm curious about your thoughts regarding some of the stuff they talked about in the second half of your talk. You had, sort of, these human agents. Basically, you want to dress a person or whatever. There's something that your simulation doesn't capture, and that's human intent. So when you're trying to dress a person, they will move in a-- and it doesn't feel like your simulation can quite replicate that. So-- We dodged that problem. I don't like to model human intent.
1:02:50
Because I don't know how to. So we really want to assume human is fully cooperative. And it understands the goal very well. It tried everything it could. Well, I guess, a human can use the pronouns. That's for human. He or she can to achieve the task. However, the capability is something that we have a pretty concrete measure for. So we wanted to measure the capability of the human instead of the intention of human. And I feel like there's more hope to do a better job with model and capability
1:03:24
versus intention. A lot of people try intention too, you know. You can do a model. You can, data-driven method. Like, for a very specific domain, maybe intention could be modelled. Oh, so sorry. What I meant was more that the pulse that you're learning for dressing a person, is there much hope of actually translating that to a real scenario? Because the person might not behave the way that you're simulating it? Right Well yes. Yes. I guess it is, kind of, related to that.
1:03:49
In that case, you know, we can't really do what we're doing now. We have to take into account intention. I didn't show you one demo that I had in a different talk, which was a robot, PR2, dressing my son. He was four at the time. And you know just basically, it failed immediately. Because they cannot understand each other in the beginning. And PR2 cannot see a boy this high. And he was jumping up and down to get PR2 to see him So yeah. Yeah. I think that's, sort of, outside of scope
1:04:25
that we're doing right now. At first, we just want to figure out how do we even model the true capability of a human? All right. I think we should continue on the other side with the reception. Oh cool. Thank you very much. Thank you.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat