Skip to main content
Loading Explore guides…
Deep Reinforcement Learning · Acedra
Home
Explore
Computer Science
Deep Reinforcement Learning
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
60 segments
Copy
Hide video
0:08
welcome back everyone so i'm really excited to continue this lecture series on reinforcement learning and in this one i'm going to zoom in specifically on deep rl deep reinforcement learning or set another way reinforcement learning using deep neural networks okay this is one of the most exciting trends uh both in control theory and in machine learning over the past decade uh and there's just a ton of new and exciting results out every month uh it seems on deep
0:40
reinforcement learning just a reminder this is um following roughly a new chapter in the second edition of our data driven science and engineering book so i wrote a reinforcement learning chapter to cap off that control theory part it was also a great excuse for me to learn more about reinforcement learning so that's something actually just a meta point for for all of you uh you know trying to explain things to people is always the best way to learn something
1:10
and this was really you know i took this as an excuse to i wanted to learn about reinforcement learning so i wrote a book chapter and made this this video sequence um so i hope you are also enjoying it as much as i am okay so this is such a big field i am going to touch on the you know the tip of the iceberg of all of the different techniques of deep reinforcement learning and i actually have another video from you know a number of months ago maybe even a year ago where i
1:37
talked at a very high level about how you can use deep neural networks and reinforcement learning and i mostly did this with examples and demonstrations of all the cool stuff you can do so today i'm going to dig deeper into more of the algorithms and how these different flavors of deep reinforcement learning fit into kind of our classic
1:57
our classic pictures that we've been developing over the last few lectures okay i'm going to jump right in and hopefully this one's kind of exciting all right so the first type of of kind of deep neural network uh embedded into a reinforcement learner kind of the simplest thing you can do is what's called a deep policy network so your policy function pi is presumably a relatively complex function of the state uh the state you find yourself in s and the action you're
2:29
going to take a so it's the probability of taking an action given that you're currently in state a and this is supposed to be optimized to maximize uh future rewards in your system okay
2:43
so this policy pie could be pretty complicated it could have there could be a lot of rules or you know physics of how you make decisions and actions optimally given the state that you find yourself in i don't know why i have a terminator here but presumably it's using a deep neural network uh to optimize its policy and so that policy function pi can be parametrized as a neural network with uh
3:08
weights theta so the input of the neural network is the state the output is some probability of which action to take and you may you know flip a coin and take an action based on that probability and so what we would be trying to do is to optimize uh over these parameters theta to maximize the future reward to give myself the best policy pi possible okay so there is a brilliant
3:35
blog by andre carpathy deep reinforcement learning pong from pixels where they essentially walk through and they actually provide a python code with about 150 lines of code that actually codes up this deep policy network so it's really really cool basically how you would take a high dimensional state s this is a high dimensional measurement of your system this is pixel space so you know there are over a thousand pixels in this image that are
4:04
input into this policy network and then essentially the policy network is optimized to make the best decision of whether or not i should move up or down my my pong paddle okay so you're this character on the right and all that you have to do is based on the current pixels decide what's the probability of moving up or down and then take an action based on that probability
4:27
and so the idea here is at a very very high level is that we have our policy parametrized by the network weights theta and we're going to optimize over theta to give us the best chance of getting a future reward and i'm being a little bit intentionally vague about you know i'm not writing down a loss function a neural network loss function that we're training over
4:51
because there are different ways that you can optimize using these deep policy networks and actually one of the uh the easiest ways essentially you can use back propagation so you know neural network training in general uses this this idea of back propagation so information kind of upstream can be propagated back and eventually used to compute gradients and optimize these these weights and so the basic idea is that i would optimize my weights by computing
5:22
gradients based on this cumulative expected future reward and so this is partly why i'm being vague is you have a lot of choices on how to represent this future reward so if you represent that that cumulative future reward using a value network so if you have a policy network to make actions to take decision to make decisions on what actions to take and you are estimating your future reward using a value network that would be an actor critic network so there's different ways
5:52
of kind of embedding information into this this deep policy network but the basic idea is that you you represent your policy as a network and you optimize over its parameters using essentially gradients through gradients of something through back propagation i'm going to talk more about this
6:11
okay um this is an aside that i think is really important and i want to tell you about but if you don't love you know math derivations you can skip these next five minutes this is kind of the guts of how policy gradient optimization works in general this is this is more general than neural networks if you have a policy function pi that is parametrized by some parameters theta this could be a support vector machine this could be some other kind of model
6:39
or it could be a neural network where the weights are parametrized by theta
6:44
then i am essentially going to derive how if you take the chain rule or if you take the gradient of this this reward function this cumulative reward function you can get a signal to update your policy weights data so again if you don't love math you can skip this next five minutes but i think this is kind of important okay so the cumulative future reward r sigma given a set of parameters theta for my policy so you can think of theta as being a policy if you like
7:14
is the following so mu sub theta is the uh kind of expected future distribution or probability of finding yourself in a state s at long times given this policy theta so it's kind of the asymptotic distribution if you played this game for a long time this would be the expected
7:36
probability of finding yourself in any of those states s out of all possible states so this is a probability distribution of where you find yourself in long times this is kind of just to be mathematically precise we need this part here
7:49
and we would add so uh so we wait by the the probability of finding yourself in a state in long time and then you weight by the quality function the quality of being in that state okay and and taking that action times the probability of taking that action given your weights theta this is a bit of a complicated expression this is you can kind of convince yourself that this is true and this is outlined in detail in the chapter in the book that this is consistent with
8:21
how we normally write this reward function as the sum of all the future possible rewards this is just folding it up using this quality function okay so it's the quality of being in a state and an action so i have to add up over all of the probabilities of being in taking any of those actions so you can this is a sum or a weighted sum over actions and similarly i have to do that same weighted sum over states and i'm using the asymptotic probability of being in that
8:51
state so this is the the cumulative future reward and what you'll notice is that we now we don't have the policy anywhere we just sorry we do have the policy but it's parameterized by theta
9:04
so essentially what we can do is take uh the gradient of this reward with respect to theta because that's the signal we're going to use to change or to update theta we're going to use gradient optimization presumably using back propagation in neural networks so we need to compute this this gradient we want to move in the direction that in theta that increases r we want to move in the direction of theta that increases the total reward and so if i take
9:32
the the chain rule here or just basically propagate this this derivative through you'll notice that this term here is a function of theta and so i compute its its gradient with respect to theta and the next step is really just uh keeping track of of algebra here so what i've done is i've taken this expression here and i divide by the policy theta and so i multiply by the policy pi theta on the left so these cancel each other out i'm just
10:03
you know dividing by it over here multiplying it by it over here so nothing changed now i'm asking myself this question and i hope you're asking the same question why didn't my mu sub theta why didn't i have to chain rule this this object here and that's a really good question
10:25
i'm going to have to think about that and maybe you can comment below in the in the video and we can work through uh kind of what went wrong here but this is this is just a sketch of how we can get uh this policy gradient information okay so so you do this and now you'll notice uh that this term here is just the gradient of the logarithm of my policy the gradient of the logarithm so now we have pi my policy times the q function
10:55
so everything to the left here is the reward is the reward that i had times the gradient of the log of the policy with respect to theta and so now this is the expected value of my quality function times the gradient of the log of my policy with respect to theta and so this can be used essentially as the the signal this i can compute uh so so essentially i can compute the log of my my uh probability pi and i can weight it by my quality function
11:30
and that gives me the information i need to update my my weights theta so i just need to compute this
11:36
using back propagation and this is better conditioned to compute with okay again this is a bit of a mathematical aside but uh if you're interested for example you know in how you actually compute this uh this policy gradient this is at least one way that you would do that So this is where I really wanted to get to is deep Q learning. Most of the most impressive demonstrations of deep reinforcement learning in the last five or ten years
12:07
many of them have been deep Q learning, where essentially you learn a quality function with a neural network. So again we know this is the basic formula for Q learning: it is an off policy temporal difference learning algorithm so this is my estimated i'm sorry this is my estimate of my future reward if I'm at state "s" now and I take action "a" now and this is my actual reward if I take that action and I actually get a reward, this is the actual
12:40
future reward. So the error signal between these, I can use to update my quality function as I learn through trial and error. So we've already had a whole lecture on this. So essentially what you can do, is you can parametrize this quality function by some neural network weights *theta* and this is really really useful because if you think about the kinds of things we use Q learning for, or deep Q learning for, the state space *s* might be astronomically big
13:09
for example if I'm playing a game of Backgammon or Chess or Go the number of board combinations in chess is more than 10^80. It's more than there are nucleons in the known universe. And Go has an even larger state space *s*. So instead of iterating over states, over and over, what I'm going to do is i'm going to parametrize this quality function by a much lower dimensional set of parameters *theta* and then optimize over *theta*.
13:37
So essentially you can think of this as extracting low dimensional features of your quality function that are parametrized by *theta* or some neural network... and we're going to optimize *theta* to get a better Q function. And this is going to really be important to address the curse of dimensionality we have in these very high dimensional state spaces. So with this one, I will write down the actual cost function involved in solving
14:06
this... so this is the neural network cost function that we use when we build a deep Q learner. Essentially the loss function that the the network is trying to minimize is the expectation of the square of this temporal difference error... so this is literally just the temporal difference signal here where now my Q functions are parametrized by *theta* and so the neural network is going to use its algorithm, stochastic gradient descent
14:38
and back propagation to optimize these parameters *theta* to give me the best possible Q function that minimizes this TD error, this temporal difference error. And again, we think that there is a strong evidence that's what biological learners are doing... at least at some level of their neurological hardware is minimizing this temporal difference error that we talked about in the last lecture.
15:03
So this is pretty cool... you can actually just take the update, turn this into a loss function and the neural network is going to optimize those parameters *theta* for you and that's been a very powerful approach. We've seen this in the deepmind atari video game playing... this is one of my favorite examples of block out you've seen this before if you watch my channel where essentially from this pixel space a deep Q learner with a convolutional level layer can learn
15:38
the right strategy, the right quality function, for what actions to take given what state it finds itself in... and it actually learns some pretty interesting things that it should actually eventually drill a hole in one side here and then it'll basically work itself out and that's the most efficient way to increase its reward, which is really cool. That's why they say human level control because this this algorithm, this deep Q learner
16:06
is exploiting the physics of the game to get solutions that only expert humans would find. and so this is the architecture so it has some convolutional layers some fully connected layers
16:19
you know from pixel space to joystick you know signal but essentially this is just a big uh kind of deep q learning demonstration convolutional cue learning
16:33
and this is just the list of all of the video games and i guess you know the ones above the line it is better at or better than human level and the ones below the line it's still not as good
16:45
okay so that was deep cue learning essentially you can have this this traditional q learning and you can cook up a loss function for your neural network that will essentially through trial and error through experience it will learn the weights theta that give you the best q function possible
17:05
and there's a variation of this that i really like called deep dueling q networks or dueling deep q networks ddqn which essentially takes your quality function and it actually splits it into two networks a value network which is just a function of the current state and an advantage network which says what is the advantage over just the value of being in that state for taking an action
17:32
a so this is a good architecture when you're the difference in in quality for different actions is very subtle so maybe i'm in a really good board position in a game of chess and the different actions i take are giving me very marginal increases or decreases of my quality function and so what it does is it splits off the value function of just being in a state from the incremental advantage of taking an action having been in that state and i mean
18:02
this is essentially equivalent to the q function it's just another way of writing the q function but now you can train two neural networks and they're kind of dueling with each other the value function is trying to get as much explanation of that q function just from the state and this advantage network a is trying to figure out what the effect of taking actions is and then you can kind of optimize over these actions and do interesting things
18:29
okay another really really important concept in machine learning sorry reinforcement learning is the actor critic learning and actor critic methods are actually way older than deep reinforcement learning so i'm going to tell you what they are in general and then i'll tell you how we use them for neural networks so the idea of an actor critic reinforcement learner is essentially to take the best of both worlds from policy-based and value-based learning
18:56
so in the you know a few lectures ago we talked about policy iteration and value iteration and they have these kind of different flavors and different benefits and strengths and so in an actor critic framework we're going to have two um two learning two learners an actor and a critic and the actor is basically trying to learn a good policy and the critic is critiquing that policy based on its estimate of the value so the critic is
19:22
learning a value function essentially and the actor is learning a policy uh okay good so the actor is trying to represent this policy pi and the the critic is going to be uh using information from its estimate of the value function and so one really really simple way to do this is
19:42
remember that policy gradient algorithm i was telling you about you can essentially do an actor critic policy gradient where what you do is you're updating the parameters of your policy uh so this is policy update data but you're using the temporal difference signal from your value learner so your value function is getting updated too you're trying to learn this value function from your critic and your critic is essentially giving you the error signal that you use to update
20:13
your policy so again this kind of mixes the best of both worlds you're using value information but you're using that to update a policy using this kind of in some cases gradient policy information
20:27
now in the context of deep neural networks there's some interesting things you can do one of the ones i like is called this advantage actor critic network and it uses that uh deep dueling queue network i told you before where you split the quality function into the value function plus the advantage of taking an action and so essentially now the actor again is a deep policy network so this is a deep neural network for the policy with some weights theta
20:54
and then the critic is a deep dueling q network uh q that tells you what's the quality of being in a state and taking an action and so here similarly this is that policy uh iteration that policy gradient iteration i wrote down earlier we derived earlier except now this policy gradient is updating the deep policy network and and gradient iteration is much much faster than the model free techniques i was telling you about in the last lecture using gradient information
21:27
will converge way faster than traditional q learning or traditional value iteration
21:35
but it requires you have a model that's parameterized by theta that you can take the derivative with respect to theta but remember in that deep policy gradient we needed our quality function and so this again this is an actor critic network because i'm doing policy iteration a policy gradient iteration in this case but i'm using a q network for that quality function so my q network is learning the quality function and i'm updating my policy using this policy gradient
22:04
network and presumably you update your your ddqn the way you always do by once you take actions and you get rewards you'll update this this q function using the temporal difference error so i think this is really cool you can kind of mix the best of a value-based formulation and really q networks are kind of value-based and policy-based you know for the optimization so you optimize the policy using value-based information so this is really cool and this is different than
22:36
q-learning which is purely value-based so in queue learning you are still updating the q function but you're updating the q function based on q information and then at the end of each optimization then you figure out what your best policy is just by reading it off whereas here you're kind of jointly you're optimizing the policy directly but using this uh this cue information this this value-based information so this is kind of a neat a neat approach okay so
23:06
that again was just the tip of the iceberg of deep reinforcement learning but we got a little bit more specific into some of these methods so very very popular deep quality function networks
23:18
you can use those to do deep policy gradients for example in an actor critic so if you like you know there is an actor critic method that uses a deep q network for the critic to do a deep policy gradient uh optimization now you'll notice conspicuously here there is this deep model predictive control block that i didn't really talk at all about it's kind of a different uh a different flavor of optimization and i'm almost certainly going to have a whole video or series on
23:51
deep model predictive control later but there's some really interesting work in the community where people are learning these kind of really good optimal nonlinear controllers using these these classic dynamical systems optimizations like model predictive control but then they try to learn a policy that mimics that that model predictive controller so this can be a very expensive online optimization it requires a lot of model forecasting it's very very
24:18
computationally intensive typically and so if i do that expensive computation maybe i'm teaching a quadrotor how to fly through an obstacle field once i learn the right control actions maybe i can embed that in a neural network maybe i can train a policy that very rapidly kind of encodes the information of these deep model predictive controllers okay so that was again just a very high level overview of some really important topics building on some of
24:44
the pieces that we've been learning over the last few lectures all right thank you
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat