Skip to main content
Loading Explore guides…
Groking in neural networks · Acedra
Home
Explore
Computer Science
Groking in neural networks
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
173 segments
Copy
Hide video
0:00
so basically at very early stage during training the train accuracy or train metric it can be something else than accuracy uh grows and then starts plateauing uh so you will think okay uh training is done but actually the test metric is still uh not at the high point so if you actually keep training much longer that you normally would then uh although the train metric still plateaus the test one will suddenly start uh increasing your network will start you know uh assembling its we together so
0:30
that it can extrapolate on uh test samples so basically the high level uh summary is that groing was observed in some very specific settings specific task specific model specific weight initialization here what we try to look at are more general setting and try to see if there is the same delayed uh property emergence happening so it turns out that it's not happening in term of train and test accuracy right those two things evolve roughly at the same uh rate and Plateau roughly at the same
0:59
time but when you start looking at test accuracy under adversarial noise then that's when you have emergence of robustness to adversarial noise on test images much after you reach this uh plateauing area of the clean train and clean test accuracy first I want to also thank imas because he's been really pushing a lot and that's all this paper got to this stage when you have a neural network which has any pwi Define nonlinearity so it can be railu can be leaky rause can be Sawtooth as well like
1:30
anything that's piecewise and linear so if you have a neural network like that with such nonlinearities then what it basically does is it cuts up the input space uh into linear regions and then linearly Maps each region to the output so basically that's what the theory says that you have this spine where the neural network boundaries are like where the spline boundaries are defined by the network weights and the mapping is also defined by the network weights a lot
1:52
like a like a an A origamy but the origamy is just not an origamy there's a little bit of stretching as well so I would say it's elas origamy that the neural network is do doing like you give it an input space an input domain and it's going to play origamy and turn it into whatever uh like some warped space and then draw a line to cut it up so the last line is going to be because the output is linear like a hyper plane in the embedding space so it's going to cut
2:16
this origami up with a straight line such that the decision boundaries separate the classes so this is basically what a neural network is doing cuz for some samples we've seen this in experiment that for some samples you would see more partitioning around it so that would mean that there's more non linearity needed there or the function is actually like more complex and then there would be some parts of space where the the cutting out of space is like more sparer so that's like a harder
2:38
region and then where there's more cutting up it's more prone to adversely attacks cuz if you move a little bit you you'd like basically be Crossing more nonlinearity so more of the activations would be changing in the network take couple of different slices 2D slices to get an estimate of what the partition statistics is like the more important thing is like estimating the partition statistics around the locality and these statistics can basically tell you how the model behaves differently for
3:01
different samples like for example if you have couple of samples uh which come from a demographic which have less expressivity assigned by the neural network you can basically say that the neural network is algorithmically biased towards those samples because it it is not that expressive in those regions whereas for other samples or a particular class it would be assigning more regions then you you can like based on these statistics you can infer like how the model behaves differently for
3:24
different sets of samples instead of like coming up with a general notion of what a neural network generally does uh this allows us to think of what a neural network generally does in different parts of region on the manifold on the data manifold or or of the manifold so these Notions I think would help us like help come converge towards like a more uh more General understanding of new network than just like a oneliner uh statement that this is what neural network does one of the basic result in
3:50
spline function approximation uh and mostly it's been studied for 1D to 1D regression is that uh when you design a spline right you have to pick out what is your part of the space and the degree of the polinomial that you use on each region and if you think about it you might say Okay pwi are fine is really not the best you just have a aine mapping per region is not so rich to represent a lot of variety within each region but it turns out that if you have to choose between well positioning your
4:19
region so finding a good partition versus uh increasing the degree of your polinomial actually it's much better to fine tune or position your uh partition according to the data even if you stay pwi are fine and just from this you can get extremely good approximation power although per region you have a really simple function and uh the way you get this power is much better than if you keep the partition the same and just increase the degree of the polinomial so
4:45
what so from this what you should take away is that uh not having smoothness and being just piecewise are fine is uh okay and actually is optimal if you can position those regions well and that's what you do with networks basically you have this adaptation of the partition when you train the parameters of your model because the partition and the aine mappings are tied together so by learning one you learn the other and the way it works in practice is that we see
5:15
most of the regions get concentrated around the training points and also extrapolated based on the rule given by the architecture and if you go in other parts of the space which is not either near the data distribution or near the extrapolation rule then you have much less region uh concentration and this is really uh important because again the smaller the regions the more precise is your approximation mlst is sponsored by sensl which is the compute platform
5:45
specifically optimized for AI workloads they support all of the latest open- Source language models out of the box like llama for example you can just choose the pricing Point choose the model that you want it spins up it's elastic Auto scale you can can pay on consumption essentially where you can have a model which is always working or it can be freeze dried when you're not using it so what are you waiting for go to Cent ml. and sign up now tab is a new AI research lab I'm starting in Z it is
6:15
funded from past Ventures involving uh AI as well uh we are hiring both Chief scientists and uh deep Planning Engineer researchers and so we are Swiss version of deeps and so a small group people very very motivated very hardworking and we try to do some a research starting with llm and Owen stey models we want to investigate reverse engineer and uh explore the techniques ourselves Professor Rand balero welcome back to mlst and congratulations on your new role at at Brown thanks thanks very
6:51
much and happy to be back uh especially to speak about like latest research uh and spine so very happy to be here you um in vented or co-invented perhaps the spline theory of of neuron networks which is something that revolutionized my understanding of deep learning I think we should just have a quick refresher so what do we mean by the spline theory of neuron networks yes so first just to put some things in a better context and uh avoid some issues legal issues later uh so splines uh have
7:21
a very rich uh Theory and they've been used probably since the 80s or maybe even before for function approximation uh but the thing is that most of the research was done for 1D 2D or maybe uh 3D input spaces uh because that's where most of the function approximation was needed maybe for like partial differential equation and things you could observe uh so what we did uh around uh 2018 was to try to understand what are current uh deep networks and at that time it was mostly convolutional
7:53
networks reset or MLPs with Ru activation Max pooling this type of non larities and it turns out that when you have this type of operation so aine operation like dense mapping or convolution then Ru or Max pooling and you keep inter living and composing those layers then the entire input output mapping is itself a continuous piecewise a fine spline so what that means is that the input space of the network so maybe the space of images if you do like mnist or C far
8:22
classification uh so this is a huge High dimensional uh space it is cut up into polop convex regions and within each of those region your network is just an aine mapping so overall it's continuous but within a regent is just an aine mapping and so you can characterize the geometry of those regions where do you have more regions than in other places and you can start to understand what your network is actually learning and why is it able to extrapolate for example or what is the impact of the
8:52
architectures regularization and thing like this there's some quite technical language here as well so I just really want want to kind of hit home to the audience that it's a bit like you know when when we train a neural network we train it to become a bit like a honeycomb mhm so so there's like this structure and a bit like a lattice maybe and and inside the holes of the honeycomb those represent um you know decisions that the neuron Network makes yeah yeah exactly uh maybe another
9:21
parallel we can do is if you do K means for example or some uh clustering method this way you learn uh partition of your space depending on which cluster your point is assigned to so you have this sort of structure with regions so if you do cin of course the mapping is piecewise constant right within the region or the cluster you are assigned to the same cluster then when you go to the next region you are assigned to another cluster so this is the type of uh Camin uh partition geometry you get
9:48
uh with deep Nets is very similar you have those regions as well but within each region instead of being constant you have an aine mapping and then you have some extra constraint that the regions are not independent from each other and they can learn even if you are very uh far in extrapolative regime but yeah the type of geometry you get is really aking to what you have with like K means or KNN and thing like this the key thing though with deep Nets and that's one of the key benefit is that
10:16
how you learn your partition or or you learn those regions is not just where you have data but everywhere in the space and that's all you get a much better extrapolation performance even in high dimension which you don't if you do like K&N for example one of the operative reasons I like spline theory is that there are a lot of people saying that neuron networks do this magical emergent reasoning and when you understand the neural network as you know Computing these spline partition
10:43
boundaries you know so so essentially it's a little bit like a locality sensitive hashing table um that's not really conducive to them reasoning or doing any different kind of computation yes so I think there is like two points from this so first as you said you have this sort of like locally sensitive Asing or template matching so you just try to fit locally what is the region geometry and the aine mapping so this you could think it's really ad do and you know like brutal rule but where you
11:12
actually have some sort of reasoning or intelling emerging is because the way you learn in a part of the space impacts uh how you learn in another part of the space even if you don't have data there and I think that's where things become from really adog like KNN or kin to something that is much more complicated and maybe much more humanlike where from one example you learn something that you will be able to reuse on an other example that you did not see during
11:40
training and that is very very far in your space so it's really uh not ad do in that sense that it's able to extrapolate from hidden rules in a very efficient way brilliant well now we're going to get to the most exciting bit today I think which is you've got you've got a paper and um you wrote this with MTS yeah and we've actually got some great content with MTS coming out as well but um it called Deep networks always Gro and and here is why give us the elevator pitch yes so yeah first I
12:11
want to also thank imas because he's been really pushing a lot and that's all this paper got to this stage and unfortunately could not be here for Visa reasons but yeah like huge Kudo to imas uh so basically the high level uh summary is that groing was observed in some very specific set things as a delayed emergence of generalization so your test set accuracy starts growing much after your train test uh your train accuracy already plateaued uh but this was in very specific settings specific
12:42
task specific model specific weight initialization here what we try to look at are more general settings like cifar so computer vision task convolutional Network res Nets and try to see if there is the same delayed uh property emergence happening so it turns out that it it's not happening in term of train and test accuracy right those two things evolve roughly at the same uh rate and Plateau roughly at the same time but when you start looking at test accuracy
13:08
under adversarial noise then that's when you have emergence of robustness to adversarial noise on test images much after you reach this uh plateauing area of the clean train and clean test accuracy so you do have adversarial groing happening in most of the settings that we found and this comes although you don't do any adversar training at all it's just as a consequence of very very long training and the emergence of those Spar Solutions where some new geometric properties of your network
13:39
naturally emerge maybe we should just Linger on a couple of things here um first of all let's do adversarial robustness yeah so what happens when a neuron network is not robust Yeah so basically uh what we do and it's a very standard way to attack a network uh you get a image an input whatever it can be something El than an image you feed it through your network and then based on gradient information you see what is the best Direction noise to add to the original input so that you can fool the
14:09
network to predict uh incorrect class so this is a white box attack you use graduent information of your network uh and this way you are able to fool the network so you can get uh for example a train set where the network on clean images has maybe 100% accuracy and just because of those really small perturbation that you cannot C by ey you can reduce its accuracy to just random guessing so it's a really really efficient attack and a lot of people try to make networks robust to that through
14:38
adversarial training so during training they actually sample those attacks and try to make the network robust to it uh and here we show that actually you become robust to those naturally just through very long training episodes basically yes now I always um looked at the spline Theory as a kind of mental intuition pump for um you know this adversarial um examples problem so we'll show a graphic on the screen now but essentially when you look at a spline partition there are just um
15:07
crisscrossing overlapping splines it's a complete mess it's very very chaotic and this is the reason why the networks are so brittle because it's so easy just to push a test example over the boundary so that the network behaves differently yeah yeah yeah exactly so basically when you see in this figure all the partition regions what you have to think about is that when your input moves a little bit and go from one region to another it's when you're mapping as a nonlinearity
15:34
and so you have a change in the prediction and so the more of those uh regions you have the more nonlinear Kings or points you will have and therefore it's much easier to pertur your network uh but when you reach this very long training stage and so this SP solution what happen is that you no longer have so many regions everywhere in the space and around your training points but instead region starts migrating away from the training point points and testing points to concentrate
16:01
near the decision boundary and now because you have much wider regions on those points it means your network is just aine within a much bigger region of the space and therefore it's just an aine mapping and it's much easier to control its sensitivity to noise so you're saying and and maybe we should actually just introduce grocking first as well and then and then we can come back to this so so grocking is this um phenomenon of delayed generalization now I know that's an oversimplification
16:26
because whenever I speak to grocking people and say that they say no that's an oversimplification but but basically you train a network for ridiculously longer than you normally would and then stuff happens later in in training can can you explain that yeah so basically uh what they did is on specific settings uh most of them are maybe constricted or simple task you can show that uh at very early stage during training the train accuracy or train metric it can be
16:53
something else than accuracy uh grows and then starts plateauing uh so you will think okay uh training is done I can uh stop my training there but actually the test metric again it can be accuracy or something else is still either near random or just very uh slightly higher but still uh not at the high point so if you actually keep training as you said much longer that you normally would then uh although the train metric still plateaus the test one will suddenly start uh increasing and
17:24
converge on its own later on which means that during training you have still some graduent information that uh make your weights change they don't have an impact on the train metric but eventually at one point your network will start you know assembling its weights together so that it can extrapolate on uh test samples okay and then there's an interesting broader story here because I know you're very interested in um the uh train you know the learning Dynamics and
17:49
there are different stages of training so at the beginning the network learns quite simple features and then as we progress through the training um stages it learns increasingly complex features now this is kind of weird though right because what you're showing is a local DEC complexification because I I imagine that things would get more complex it's learning high frequency information but but the network is learning a local DEC complexification to kind of stretch out
18:13
those those boundaries why and and then there's a question of well what learning signal is it using to do that yeah yeah yeah exactly so basically as you said you have this two stage training a bit like a in a double descend Dynamic where during the first stage you start from random initial ation and your network focuses aggressively on the points use a lot of regions around them so it's sort of memorization so it's still able to extrapolate but still it's really really
18:40
focusing on the actual uh points so lots of Regions near them and uh very smooth mapping uh and no simplification anywhere in the in the space but then once you reach this stage and you keep training again like much longer than what you will uh usually do what happens is that there is still some gradient information uh that is conveyed through your loss and the network will as you said DEC complexify around those points and so the regions will start to migrate and move away and instead they will
19:10
focus on where the decision boundary is so all the allocated parameters are trying to fit the decision boundary very precisely instead of remembering where those points are and because of this the radius of the regions will increase around those points and therefore you will get uh adversarial robustness and you will reach this stage where your network goes from something really smooth almost like uniformly Smooth to being peace wise constant which is what you should converge to in the limit
19:40
theoretically so this is a really nice property that naturally emerge although you have to train for very very very long time for it to happen yes yes I suppose you could argue it both ways whether it's complexification or DEC complexification but but the remarkable thing as you say it moves from looking at the training examples to looking at the regions and there's this great figure that I'm going to show again on the screen now where um after this kind of um grocking phase has happened you
20:08
get these these partitions emerge and it looks a little bit like a topology map or like a contour map or something like that and these partitions look very much like a veroni diagram so so it looks like the partitions are equidistant the boundaries are equidistant between the points and this it looks like a mountain structure it's not a single like decision it's actually many that have been squashed together yeah yeah yeah exactly so when you see those huge concentration of partition regions all
20:36
of those are squashed together around the decision boundary to bring the uh representation capacity to go from one class to the other uh and this is what you want right because when you are near a point and it's in neighborhood you just want a simple aine or even a constant mapping you don't need to put a lot of uh parameters there a lot of regions but instead where you want to put all your is when you are at the decision boundary because this is where you need to go
21:02
from one class to the other so you need actual curvature in your mapping so this transition is really the key that goes from uniform smoothness in the space to piecewise constant mapping and this is what brings adversar robustness as well and again this emerge only after very very long training because as you said before it's not the first thing that the network learns right so it's sort of a edden solution that emerges after very long training because maybe the gradient
21:29
Norm to reach there is very small or maybe because just fighting the implicit bias of your architecture uh because as we also showed uh as a function of the strength of the regularization you can control the rate at which this emergence happens so basically the more you regularize with things like qu Decay the more you fight this SP solution and therefore it will maybe not happen at all or happen very uh late during training even later this is um it's related to sparcity right so we know
22:00
when we do iterative magnitude pruning and you know we we train a dense Network and then we kind of like you know we we pull out all of the um the the weights that have low magnitude and and the the sparse Network strangely is more robust even though we've taken away most of you know most of most of the weight and isn't it interesting that after this grocking phenomenon the network we get resembles a sparse Network yes yes so it's exactly related so in another paper
22:25
we showed that you can actually do pruning in a reined way uh as a way to simplify the partition to make it focus on the decision boundary rather than the point and you can prove a relation between prining the impact on the geometry of the partition and even other methods like collapse in the rank of your parameters and uh this type of regularizer so basically there is a onetoone correspondence between all those things and the nice thing is that the partition gives you a single
22:54
geometric object where you can visualize all of those and understand where and why they are beneficial but yeah exactly it's very related to like this lottery ticket hypothesis and like iterative printing in a way when you do itative prining what you probably do is you first switch this like complicated solution with like uniform distribution of the region but then on your own aggressively you remove most of them that are uh useless for your task so probably the one near your points so you
23:20
bring the model closer to this later stage but through active uh prining of your model parameters or units how is the phenomenon related to double descent yeah so this is uh one of the thing also related to neural collapse that we mentioned before so basically uh there is this Dynamic that at first there is this sort of uh memorization and then you start to uh learn to extrapolate except that here we don't look at it in term of capacity as number of parameters
23:48
it's more uh geometry of the partition but you have the exact same thing so if you look at uh this local complexity measure that we derive you can see the Dynamics or the migration of the region so when you look at the beginning of training uh and when you start having an increase in train and test accuracy you have a lot of regions that get concentrated near the points and this is like the first Ascent that you get after the first descent and this is where you
24:14
get no robustness at all so this is where people will start will stop training because your train and test accuracy look good uh but then you don't get any robustness if you keep training then suddenly the regions migrate away so you see this second descent happening and this is when eventually robustness emerged because again those regions moved away so the ones near the training points and test points have bigger radius and therefore you get robustness but this is uh giving you a new way to
24:42
look at things like double desent or just uh training Dynamics through the length of the geometry of the partition which is really nice because uh until now most of the things that were studied was looking at the loss function or train and test accuracy uh which are may be very like task specific or uh very like black box in the sense that you just look at a network F and you don't really dive into it but now you have this new way to look at what's happening within your network in term of this
25:11
partition and so one of the big future question is to try to rederive most of those result in term of the uh geometric properties of the partition right now um we use regularizers because um I mean certainly this was more the the old school view but by kind of deliberately straining networks to be simple they they train better but what we're talking about now is that actually we want to have um a type of complexity so I think you said to me earlier that if we're not
25:40
careful about our regularizers we might not even get this effect in the first place yeah yeah exactly so uh so first of all there are many ways to regularize a network like implicit and explicit regularization and even methods like bash normalization for example that we may think okay it's just a way to uh make training uh easier because you have normalization actually it acts as a regularizer as well so in this other paper we show that if you have batch for example you actually actively try to
26:05
concentrate the Regions near your training points so when you employ techniques like this you will actually fight this sparse solution and therefore you cannot get there at all or maybe you can get there through even longer uh training episodes and as you said when you use other things so like weight Decay for example this has also a strong bias you say Okay a solution near zero is but brings your model to a nice solution uh and usually all those regularizer try to enforce smoothness in
26:35
your mapping in the L2 sense of it but because of this you don't try to get to a piecewise constant solution which is the one that will give you adversary robustness and which is the one that you eventually reach When You Reach this adversar working phase so that's why there is also a lot of implicit biases that we put through standard regularization that maybe we need to rethink if you want to speed up the the emergence of adversary gring for example one of the themes that we'll talk about
27:02
today is that you know new networks do incredibly interesting things later on in in the training Dynamics and this is great right if you're meta or if you're Google or something because they've got these big GPU clusters and and they they train neural networks Beyond which you know is it's it's out of our reach for for the normal person so so we we want to we want to capture that behavior and we want the neural networks to do it earlier and I know you've already done
27:25
some work with um I think didn't you build a geometrically inspired regularizer that kind of made the um the boundaries orthogonal to each other could could we design a regularizer that would encourage this gring Behavior sooner yeah yeah so that's a very good question so first as you mentioned uh so right now we just we reach this adversary robustness through very long training and because we don't actually do aders training the robustness we get seems to be uh very good across
27:51
different type of aders attack so that's a really beneficial property that people want but as you said the problem is that because you need to TR for so long right now it's not a solution that everyone can get access to and that's a really big limitation so that's why when big access of research is how to speed up the emergence of of adverse gring so that everyone can get access to it uh and then to your second Point yes so there are a lot of ways we can build regularizer based on the geometric
28:19
understanding of the partition so for example one thing that is really easy to compute even for very very large network is the distance from a point to the nearest uh boundary of the region it lives in so this uh quantity this distance is differentiable with respect to the parameter of the model and it's really fast uh to obtain it which means that you can actually use it as a regularizer during training so this is just one example but there are a lot of ways you can derive a differentiable
28:48
regularizer that uh can be used to enforce some constraint in your partition another thing is that you could actually not use a regularizer but try to build the architecture with those constraint and force so one typical example is suppose you have no biases at all in your network then the type of partition you get is constrained to be Central so all the regions are cones going away from zero to infinity and this is the art constraint so if you don't have any biases then you
29:16
constraint your partition to be always look like this and therefore you don't need to have extra regularizer so there are a lot of ways you can either build art constraint into your architecture or parameter ation of the weight or build a different regularizer that you can use during training yeah absolutely I mean after speaking with Sarah last week as well she had this paper out about the um you know the EU AI Act and the executive order and and they had this um hard
29:43
absolutist limit on the number of compute flops which is a tally of the amount of computation operations that that that we do and the thing is we're in the regime now where um people just think oh you know there's a commensurate relationship between capabilities and compute so let's throw more compute at it and what what we need to be doing is smart flops but but further we need people like you that actually have a theory of neuron networks because had you not had this spline the of neural
30:07
networks it wouldn't be possible for you to have the mental model and to design new regularizers in a principled way it's so important yeah yeah exactly I think one of the key benefit of using splines is that not only it giv you some theoretical uh guarantees and solutions but also the type of things you can visualize are really easy to interpret so even if you never used splines before it's really easy to look at those figures of like the partition the regions and see why this partition is
30:34
better than this one for this specific task or in term of robustness so uh the good thing is to not just have a theoretical understanding of deep networks but have something that any nonexpert can visualize and act on and use to actually do better training of state-of-the-art model so you want it to be tractable interpretable and easy for everyone to to get on board with basically and as you said uh from this you can get really strong insights for example through this paper uh we see
31:01
that if you have uh allocated number of flops uh the common wisdom or at least what most of the people may do is just say okay let me try to Feit the biggest model uh that I can on my GPU and then whatever remaining amount of flops I have I will just use it uh for training time but from this paper what you see is that actually depending on the type of application you want if you want for example adverso business you may want to use a much smaller model but allocate
31:28
flops for a training time instead and when you do this different flop allocation as you said like smart flop allocation then you will get a model that is adversar robust uh after this long training episode so that's why through this you should bring back questions about okay where do you allocate your flops based on what properties you want your network to have talking about your results a little bit how does it change depending on you know the type of problem the the data set and
31:54
so on what what have you seeing yeah so uh there are a few trends that we saw so for example as the size of the data set increase or decrease or also as a function of the noise you have in your labels then the emergence of gring can be delayed or not so those are things to uh keep in mind as well and those are things that could be used later to also try to speed up groing so for example if you have maybe a smart curriculum for training or if you have a sort of like
32:18
teacher student uh training that's the type of thing you could try to do to speed up the emerence of gr King and now there are more and more paper trying to look at this because uh again all of those things are intertwined so if you think back again about the partition you have to think about the partition as being adaptive to the data set that you have in hand and that's one of the beauty of uh the splines that deep networks use as opposed to standard spline is that now the partition really
32:45
adapts through gradient Descent of your weight to your data distribution uh and the loss that you have so if you change all your points are distributed the number of points the dimension in which they live or just the parameterization of your network it will impact the partition that you learn and therefore will impact the geometric properties that are either beneficial or not for you so all of those things are intertwin and the the beauty of the spline uh interpretation of deep net is that you
33:11
can understand this relationship precisely and uh you can like provable guarantees on why you get this partition and not another one so now you can have informed decisions or to act or to parameterize your model to reach a state that is good for your Downstream task yeah very interesting I mean I think now might be a good time to talk about your um local complexification measure the only reason I bring this up is the first stage is we can you know when we've got
33:36
this measure of complexification and grocking we can change the hyperparameters the you know like the the prediction architecture and so on to optimize this effect I mean maybe Downstream we can actually have a principled way of Designing the architecture to to optimize grocking but how did you design that that complexification yeah yeah exactly so basically uh what we are trying to do is look at how many regions or how complicated is a partition near a specific point so a good proxy for that
34:03
is simply to count the number of regions that are nearby uh and as you mentioned before this is roughly equivalent as how much bits of information your network has in this neighborhood and basically uh what we do is a proxy to get uh that number of regions very quickly even if you use very large model uh which is based on per layer counting of the number of vertices that are in the Epsilon bow uh and then we do some abls to to show that this is a good proxy for
34:29
the number of regions uh but this is what we measure and basically what you see in those local complexity graph is that this number has different Dynamic during training and the very interesting thing that you observe is that it picks when the train and test accuracy reach the plateauing phase but then it starts decreasing much before you get the adversarial groing uh happening which means that this metric is actually much better it's more sensitive to training
34:58
Dynamics and what happens in in terms of your deep Network geometry because this metric starts to show this as you said before the complexification around the point much before you get the adversarial groing so this is really nice because you have a sensitive metric uh that tells you all the geometry of your model evolves and this also opens a whole new door to for example new ways to do early stopping or to understand where your network is done training or not because even if you are not
35:28
interested in this adversar groing so let's say you just want to know okay when the first stage of training happen you can just look at this metric and start to look okay when is it plateauing and when you reach this first Plateau area you know you can stop training here Al you never computed train or test accuracy so you see that uh a proxy proxy metric that characterize the geometry of your partition is actually all you need to re understand the stage in which your network is and do you need
35:55
to stop training or do you need to change some IP per parameters or architectures or thing like this uh and in fact that's something that some people have been using before so there is uh this paper by Atlas from ostin where they show that you can do neural architecture search just by looking at some statistics about the the partition so this is informative enough to do neural architecture search uh and get a good uh good model to train on interesting why do neuron networks kind
36:23
of learn low complexity low frequency features first yeah so this this is a a good question so there is a lot of work on this like implicit bias or like Simplicity bias uh where you just try to learn the simplest trule first like those spirous correlation things right and that's one thing that people are trying to fight uh a lot because of course you think you learn the good solution but actually you just learn a shortcut solution that maybe will put you in a really bad situation later once
36:48
you deploy your model and that's something that is a really active area uh so there are some cases where we can explain why this is okay so for example if you learn by reconstruction right there is this other paper where we show okay uh there is this bias that comes from the data set because this uh Simplicity bias is transferred in term of okay which frequencies of your image have the most gradient information and it turns out those are the low frequency part of your image and you can prove why
37:16
this is the case when you learn representation by your construction uh but in the most general setting I think this is still open question and intertwining uh the implicit bias of your architecture the way we do training and maybe even other things like bat normalization uh and that augmentation as well and you implied that high complexity features are less likely to be shortcut features what's the intuition for that yeah exactly so basically uh at least for perception
37:42
task for example uh you can show that when you look at high frequency features visually you have much less of those perous correlation for example between what is the background and what is the actual object in the image because this information is not present in the high frequency part of the images so uh naturally because you remove them then if you train on this type of filtered images you will of course remove the opportunity for the network to learn those shortcut solution and it will have
38:08
instead to focus okay what is the actual shape of the object I try to uh classify instead of just okay is the background grass or is the background nice Beach and therefore you can just from this tell what is the object in the image so this term emergence comes up quite a lot as well it's it's a b of a wooly word and people talk about it in respect of grocking and sometimes it's overestimated because you know like they use log plots and the grocking isn't quite as transient as people think it is
38:37
but does it make sense to use the word emergence uh to me it does because uh it's a phenomenon that happens on its own so it's not like uh after you reach this first stage of training then actively you change the learning rate or you change the regularizer and you make it happen you don't do anything you keep training and suddenly uh this new new property emerges on its own so I think to me the emergence term is actually quite fit for this as long as you don't
39:04
actively do something to make it happen on its own basically very cool so I'm looking at this diagram again which shows you know the kind of the honeycomb the topology map and this is quite a good example because it it shows a real clean partitioning between the examples but I wondered are there any more complex examples and and in the future might there be a kind of meta partitioning scheme that kind of like you know cens it even more yeah yeah yeah exactly so
39:28
from this understanding we know what type of partition we're trying to get to right so the next question is okay how can we try to unforce that into your network and is there a way to either impose that through the parameterization of your model or through regularization or through printing as you mentioned before and I think little by little once we understand okay what is the geometric property you need or does it transfer in term of parameters of the weight now we
39:54
are at a stage where we are able to say okay we can actually derive a method to reach to that stage actively and earlier during training and this again through the spline partition is much more intuitive than if you just treat the network f as a blackbox model right because you look at those image you say okay we want to increase the reduce of your region you know for example we can pinpoint which units are responsible for each uh partition boundaries so we know okay which one do
40:21
you need to print to make the region bigger or not so all of those things uh are tied together and because we have this understanding it becomes much easier to actually act on the network to reach that solution faster amazing now just for the folks at home I can't impress enough on you folks how how big of a result this is this is absolutely brilliant so you know I've had loads of adversarial um robustness researchers on like Nicholas Carini I've got um you know um Andrew ilas tomorrow and for
40:49
years people said that this is an intractable problem that anything you do uh to fix the robustness doesn't actually fix it and you just reduce the headline accuracy and you've actually kind of proven that you're doing it in an optimal way I mean maybe can you explain that but what do you think this means just for the whole Space yeah yeah so this is very interesting so I think there are many question into this one uh so one thing first of all is that uh here uh as we mentioned before we don't
41:15
use strong regularization right so maybe a lot of the previous results were under a strong regularization setting then that's where maybe it's impossible to get Universal robustness or it's much harder to to get to it so here we show maybe a new way where uh people could try to uh look at this problem differently uh but also the nice thing here is that we don't do adversarial training right so because of this we don't really overfeed the robustness to a specific type of attack uh and in a
41:42
sense that's also uh uh link with like overfitting if you train with adversarial examples you overfit your robustness to this specific adversar attack and maybe just because of this overfitting mechanism you become more sensitive to another type of attack but but here we don't do any adversarial training this robustness comes naturally and therefore there is no reason to think it overfit to any of the specific attack because it was not used during training right so I think because it's
42:08
uh implicit emergence on its own then by definition it will be much more Universal than the active way to get adversary robustness that people have been using before so I think this opens a new door to try to uh look at those results again and see if there is maybe a new way or a new compromise you can get through this uh implicit uh emergence amazing well um congratulations on this amazing work and and also um my my love goes out to tnas you've both been driving this work for
42:38
so long now and I'm so happy that you're finally you know proving to the world how incredibly important the spline theory is yes yes yes and also the nice thing is that all of those results are not specific to Vision or specific architecture as well so that's one of the beauty of splines is that uh your network is a line regardless of the data modality regardless of the input dimensionality so whatever Insight you get really transfers across a lot of uh applications and so when you derive
43:08
something new or you have a new paper you don't solve just one problem but a whole family of problems so that's also one of the power of this sort of like theoretical understanding is that you can do one proof and it will useful to many so it's really efficient it might just be worth saying as well that people might think oh you know this the spine theory is just talking about MLPs but I just want to impress upon folks that every neuron network is an MLP technically right you know whether it's
43:33
a Transformer every self- attention layer has an MLP whether it's a graph convolution on your own network or whatever it's it's all an MLP which yeah yeah exactly if you think about the convolution you can just think of it as a MLP with a circulant blood circulant Matrix so just a constraint parameter but still you have the same the world network is inter leave of aine mapping and nonlinearity and this is true for all the the current architectures and in fact in this other paper the polarity
43:59
paper where we control quality and diversity of generated sample we show that you can use the spline formulation and spline results to improve state-ofthe-art results on huge architectures uh so this is not something that is only for toy example or small dimensional setting this is really something that can give you answers and actionable solution for state of thee art models across modalities very cool very cool okay so we're going to move on to your next paper
44:28
um so your second paper is learning by reconstruction produces uninformative features for perception so so this is this is talking about the difference between a reconstruction like an auto encoder where where you're actually reconstructing an image and then you're kind of like you know doing the mean square error between the original and the reconstructed or the other way of doing it is so-called like contrastive and non-contrastive models where you actually look at the difference in the
44:50
latent space yes correct tell tell us about this paper yeah yeah exactly so in this paper uh we Tred to give some answer and some explanation to some phenomenons that were observed empirically uh and two of them are if you learn a representation by reconstruction the representation you get is a good Baseline but it's not state-ofthe-art and you need some fine-tuning to really uh bring its quality ey for the specific Downstream task that you are trying to solve and
45:15
the second observation is that uh the quality of your representation to solve a task does not align well with how good the reconstructed samples are and often even if the reconstructed sample look good by ey you still need to keep training for a very long time for the representation to become useful for uh perception Downstream task so those two observations have been known for a while uh and the point is can we try to explain why and maybe from that try to derive better methods later on and the
45:45
main takeaway is that because reconstruction method in input space so in pixel space for images use mean squared eror most of the gradient information come from the low frequency of the images and those uh features are not the one that are useful for perception task and there is this nice example right where we look at okay what information is unced in the low frequency features and in the high frequency features and you can easily see even You by ey that okay the low
46:13
frequency ones are not enough for me to tell what class is this image from but the high frequency one is enough so basically this bias that comes from just the data set distribution and or uh it's against spectrum is is something that is uh copied in the auto encoder but because it's bias is not align with the downstream task you get a suboptimal representation yeah I mean I'm going to show the the figure on the first page now and it shows this um igen spectrum
46:42
and on the right hand side it shows the features that the neuron Network learns first and they have a bigger Mass so they kind of Dominate and then later on the the higher frequency features are are learned can you kind of and but there's a couple of examples as well so the image on the left is the high frequency features which is very recognizable the one on the left is the low frequency features and it's just blurry exactly yeah so as you said in this uh graph where we see the uh
47:07
distribution of again value so you see on the right side you have the ones corresponding to the low frequency one which are the one with the highest again value that's where you have most of the energy in your image and you can show that's what we do in the paper that this is what is going to dominate the gradient information and therefore that's what you're going to naturally learn first this is what gives you the biggest reduction in term of mean squared error so in a sense you could
47:34
say if there is one frequency that gives you the biggest reduction in me squ error which is it the answer is it's the low frequency one so because of this and because we do gr and descent obviously we are going to learn this one first and it's just natural because we just try to copy the bias that is already present in the data set so we learn those ones first and then if you train long enough and if you have enough capacity in your auto encoder you will start learning The
48:00
High Frequency details which are the ones with much lesser amplitude so much less uh gradient information so only then you will learn features that become useful to solve perception task because those are the ones that even by I you can see okay they contain the features that can tell me okay this image is this class or this class very interesting now um you said there is there's a new method now you know people are starting to use the um the Reconstruction you
48:23
know something like an auto encoder just because it's really easy and people can do it at but it does it does pick up this this sort of data set bias because many data sets do actually have you know they're dominated by these these low frequency attributes but it is data set specific and you can actually add noise to to fix this yeah yeah exactly so basically it is data set specific because as you saw in this figure you try to mimic the bias in term of like EG spectrum of your data
48:48
set but this uh against Spectrum will be different depending if you have for example background or no if you have different translation uh in your images how many classes or objects you have so this bias or this misalignment between reconstruction and perception feature is data set specific and the simpler the data set like mnist svhn the more aligned are those two task because in a sense every bit of information in the image become useful for both reconstruction and perception because
49:17
there is no background no resons variable no noise so if you learn to reconstruct you learn to recognize basically but this is not the case when you move to really realistic images so higher resolution colors background many variety of objects like imaginet then that's where the alignment uh becomes really really bad so this is data set specific and as you said so now for example people use a lot moton encoder which is a different version of the noising auto coder with a different
49:45
noise strategy so in this case you don't just take as input to the auto coder the original image and try to reconstruct it instead you add some noise some perturbation to the original image and then you try to reconstruct the original image so you try to Denise uh the noise distribution that you you are using so in denoising Auto coder it was common to use additive isotropic Goan noise but now in moton coder you actually mask big Parts Big Blocks of the image so you see
50:12
you have a different type of noise strategy and what we show in the paper is that you can play with this noise strategy to try to cter the bias that you have in the data set so in a way you try to say the network okay I know you try to mimic this bias but let me make your life harder for this part of the spectrum or those type of features so that instead you just focus on the other side which probably is better for my Downstream task and so that's why through careful tuning of your uh noise
50:39
distribution you can try to realign the learning bi construction and learning for perception features but still it's a really really uh active process and in a sense how can you do this process if you not have access to labels a prior right so this is a big question and that's why one of the future research is or can you automatically design a noise distribution yeah exactly because you were saying earlier that I mean for example something like pink noise which I which I think has a a log power
51:06
Spectrum if if I understand so you can actually design the um the noise Spectrum to kind of preferentially focus on you know let's say the low frequency features yeah exactly so for example uh if you tell me okay for my down stream task I know what type of frequencies I should focus on and which ones are useless then from this information alone so you don't really need labels but still you need this form of like quick supervision right but from this information alone then you can reverse
51:32
and generate and figure out what is the noise strategy so that when you learn a represent a representation by reconstruction you will not uncode the useless features and only focus on the on that are useful for your Downstream task but again this requires some a prior expert knowledge and this is not something that is always so easy to do so here for example for perception we show that okay it's actually quite easy if you focus on the more high frequency details instead of the low frequency one
51:58
you get better representations but if you have another Downstream task tomorrow like depth estimation or trying to count the number of trees in an image it's not really clear if it's just High versus low frequency so in general depending on your do stream task it may be hard to really Define what is the noising strategy that makes most sense to you and also you need to be able to uh implement this noising strategy right because if it's too complicated and too
52:21
involv and it slow downs your training a lot then it's also not something you can use in practice I understand so so the punch line is if you use a reconstruction loss you're basically inheriting a bunch of data set biases which are going to like mess you up down Downstream therefore we should use these um you know kind of contrastive um methods some of the audience might need a refresher on on what that means how does how does that work yeah so uh in most of those what I would say
52:44
reconstruction free self supervised learning method uh You observe different views of the original image so this can be because you apply uh different data augmentations to it or you extract adjacent frames in a video or you have just different viewpoints of the same building and then what you are trying to do is to get those uh input image through your network uh and compare their representation in the edding space and have them get the same representation for all those different
53:10
views so this uh is also a form of comparison but it happened in the edding space instead of trying to reconstruct the original image and compare the Reconstruction to the original input and is the intuition there as we were saying before that a trained neuron Network on after it's reached maturity it tends to focus on the high complexity information so you get that for free when you're comparing the latent space you're actually kind of focusing in on the type
53:33
of representation that you want yeah yeah exactly so because you work in edding space then you allow to disregard a lot of the details about the input image that you don't need so of course this is a function of how you define the data augmentation and how you do this like positive pair sampling but it's much easier to disregard things because you are not trying to compare to the original pixel space image in term of mean Square ER so just because of this you are able to disregard useless
53:59
information and you are therefore able to control much more easily what features your network is focusing on instead if you try to do reconstruction the only way you could say okay I don't want to focus for example on the leaves of the tree is to try to come up with a new loss that becomes invariant to that but this on its own is a huge research program and maybe there is no easy solution or at least no tractable solution so that's why working in edding space is a really really nice proxy and
54:27
efficient proxy where you can keep using this mean squ eror but in this new space where you can easily disregard information about the input brilliant third paper characterizing large language model geometry helps solve toxicity detection and generation yeah give us the elevator pitch great so there are two key uh components in this paper so one which goes back to the spline again uh where we look at a single layer of llm and you can see you can decompose it into two big blocks you
54:56
have the multi-ad attention and then the following MLP block so this is true for each of the layers of most of the current llm architectures so if you look at the MLP block alone of each layer you can interpret it as a spline again whether it's using R activation or switch activation again is the same thing under this spline Viewpoint and you can try to understand okay can we characterize the region in which a given prompt Falls in so again pure geometri geometrical characterization is the
55:26
region big or small what characteristic geometric characteristic this region has so from this we derive seven very simple features that characterize this and so it's seven features per MLP block so it grows inly with the number of layers that you have but the total number of features is really small for example even for a 70b model you have like about 500 features that fully characterize a given input prompt uh and then what we say is okay are those features
55:53
informative at all about the prompt and what we find is that even if you do some very simple for example TSN visualization so really like unsupervised dimensionality reduction into 2D and we say okay what is the distribution of those features you can see that they are clustered for example based on the modality of the prompt like which data set this PR comes from is it uh about mathematics about law uh Medical Data uh so just based on this naturally those features are already
56:21
cluster in term of prompt modality so that's one of the thing you get uh naturally and also for example for toxicity detection you can also see that if there is toxicity in the prompt or not you get different clusters of those features so this is really interesting because it shows that again by characterizing the geometry of your partition and of the region in which a prompt Falls in suddenly you can already have a really strong characterization of what your prompt is about so and again
56:51
this can be applied to any pre-trained llm uh does not need to have like expert knowledge or anything and it's really easy to extract them and use that to do like different Downstream tags that you may want to do you can feed them as input to a linear layer if you want to do something else than toxicity detection you can just uh try to train a model to predict whatever quantity you want from those features and you will get a very good Baseline yeah so this is
57:15
another great example that I think a lot of people now when they do um you know unsupervised representation learning they're they're looking at the the space they're not looking at the geometry you know like this partition boundaries so another great example of how the spline Theory really really helps us here so you've created a whole bunch of features that derive that describe statistically describe this geometry so one of the features might be what's the you know
57:38
the average distance from the boundary or or something like that exactly and these features alone so you do tisy on them you see they work well but you can just build a linear probe so just a simple linear classifier or random forest or something like that and these are significantly more informative than everything else out there yeah yeah exactly so once thing we compared to was uh to look at for example hugging phas the most downloaded models for toxicity detection and we compare those uh to
58:04
this basine as you said where we just extract those features and we train a linear head on top to do like toxicity detection and we see that uh not only we're able to do the prediction with as low latency as them but also we get a better detection rate so this is a really competitive solution uh and of course this is really nice because you can reuse a pre-train model and you can control or you use those features for your Downstream task so here it was toxicity
58:30
detection but you can really see you can just fit it to anything that you want uh and also as you said the benefit of this is it's very easy to derive those features because suppose you don't know anything about your model uh each layer has representation which can be like 4 10,000 uh Dimension you have this at each layer for each token so you don't really know how to make sense of that and you cannot just say okay I'm going to extract all the representation and
58:54
concatenate them together because have like a million dimensionals uh representation of your prompt which means that if you want to learn a linear Probe on it you will need to play a lot with regularization feature selection so it's going to be really cumbersome to you but here because of this PL intuition we know how to derive uh most informative features that characterize your prompt therefore we get only a few hundred of them and therefore you can learn a linear probe even if you have
59:20
just a few samples for your Downstream task yeah that's incredible I mean I'm looking at your results now and I think the the most um famous model for toxicity detection this is on the Omni toxic data set so this model Martin Hart's been downloaded 1.2 million times I think in the last month and that had um like a you know like a rock the area under the curve of of around you know 73 and a half% yours is just on on the linear uh probe um on on llama 27b um 99.18% and your latency is like
59:50
commensurate with with with the best one yeah yeah exactly and the good thing with this method is that because per layer you extract the features you can actually control the tradeoff between latency and accuracy by using for example in this case only the first three layers features uh so you get a good latency and still a good accuracy but you could improve the accuracy by using more layers and increasing the latency or if you wanted to even further reduce the latency you just use the
1:00:15
first one or first two layers so maybe you will lose a bit of accuracy but you will further decrease the latency so again because you can control how much features you want to use you will have a trade-off as opposed to current Solutions which are okay I'm going to retrain a new llm and treat this toxicity detection as a new task for LM and therefore you need to use it as a sort of blackbox detector yeah I mean I I think Folks at home should be looking into spline statistics I mean you know
1:00:42
for example there are loads of people you know anthropic did this um sparse Auto encoder on language model representations for Claude Sonic and they did you know the the San Francisco bridge thing and they they they were just using like you know the vector space like if they use spline features it would be even better yeah yeah exactly and the nice thing with uh this interpretation as well is that you can do much more than just toxicity detection so as they did in this study
1:01:07
you can try to do this for example for data filtering or to try to derive okay which prompts should you use or not for training or to compare models even right you could try to create a new llm that is sort of orthogonal to the current one in term of those features so there is a lot of things you can do because also those features are different which means that you can actually use them during training so suddenly it opens the door to to many things and those features you can compute them very
1:01:34
quickly on the Fly which means that you can use them from the getco at every step as extra regularizer as as extra training objective yeah I think that's really important because we were talking earlier about you know let's say building a new regularizer but as you say the features are differentiable and you can you can use them um in many many different ways so you know interpretability robustness spess like many many parts of the training Dynamics can use these features yeah exactly and
1:01:58
so you can use them as regularizer but you could even use them to try to derive adversarial attacks uh so going back to the first topic because since you can differentiate through them you could try to see okay can you use that to manipulate the prompt to for example make it look more toxic or less toxic or you can do a lot of things because you get differentiability uh so this is a great great property to amazing okay so the the second part of this was you were
1:02:23
looking at the intrinsic Subspace with to the the promt can you tell us about that yeah yeah exactly so this first part was about trying to understand the MLP block and see what we can do from this geometric understanding the other part of llm layer is this multi- attention right so here we try to understand okay what is this characterizing geometrically the given input prompt and so we derive this nice uh proxy scalar which is the intrinsic dimension of the space in which the
1:02:49
prompt lives in and in short you can derive this as a function of the sparcity of the attention that you get and from this we can really easy see really easy see that okay current training uh promts have some intrinsic Dimension distribution and therefore we can try to create new prompts with increased or decreased intrinsic Dimension and so of course a PRI you might think okay this is interesting but what can you use uh this for and one application we found in the paper is
1:03:16
that if you actually increase artificially the intrinsic Dimension to look like uh a point which was far from the training data maybe that's a space where your llm was not R shft so you can bypass the rft mechanism and make your llm generate toxic concers and in fact this is really natural because the way you prevent toxic uh generation is just during training so you try to say okay don't say that here don't say that here but do you extrapolate that was the open
1:03:44
question and here what we show is that once you explore a new part of the space that was not used uh during training through this manipulation of the intrinsic Dimension you can make your uh RF llm generate toxic answers although if you use like normal original prompts it will say okay I cannot say this because it's a that's something I'm Ed to say yeah this is so interesting so so there's a real theme here that we've been talking about that there is a kind
1:04:10
of spectrum of of complexity um in neur network training Dynamics and and representations and you've got this figure here where you show there's a commensurate relationship between the context length and the intrinsic Dimension and I think a better way of kind of in explaining that is is the complexity of the representations now rhf has a kind of complexity um limit at the moment which means it's only capable of addressing low complexity representations or things with a small
1:04:38
intrinsic Subspace so when you have a really long context length this is related to the interview I did with the University of Toronto students the other the other week they've got this self- attention control ability theorem and they basically say that as the context length increases the control ability increases which means you can make a language model say anything and that's to do with like your moving you're like increasing the the the intrinsic complex the complexity of the representation
1:05:01
yeah exactly because if you think about it uh the higher dimensional space you are working on the harder it is to control what your llm is going to do or not because it means the number of samples you will need to really control it grows exponentially right unless you have a smart parametrization on how you do this uh parameterization so that you can learn to extrapolate from a few samples but that's not the current way to do this fine tuning so as increase the context length and also as you
1:05:28
increase the uh related concept between token so that there is really a nonsparse uh self attention mask then you go to a part of the space that was never seen before and that is really really high dimensional and so it's less likely that uh people who created the llm were able to control what's happening there and there because we are just in this very high dimensional space where no one can guarantee anything at this point yeah now now this is this um is a real problem for kind of steerability
1:05:56
alignment interpretability and so on I was speaking with Nora belrose the other day and she was talking about concept scrubbing you know so you can scrub Concepts out of neural networks and it works really well early in training but as the as the network complexifies then it kind of adapts and and it counteracts any concept scrubbing that that that you do so we have a real problem that as neuron networks complexify we can no longer control what is going on yeah
1:06:19
yeah exactly and also it's really hard to find a solution to it by our self acting on the data because based on our vision and the way we learn there is something where we can think okay we remove this information from the data and we think that's all we need to do but actually because there are so many right we in high dimensional spaces so many things happen between different input Dimensions maybe there is still this concept embedded in other part of the data and the network is going to be
1:06:45
able to find it if if it's a nice Like Short Circuit Solution for it so basically uh that's where we have to be really careful about our intuition through visual inspection and two dimensional reasoning may not scale when you go to re High dimensional spaces because a lot of other things are going on and the network will pick up on those so that's where there is a huge need for provable Solutions because you need to understand what the network is trying to learn or to control it geometrically to
1:07:12
prevent from learning those uh shortcut solution and also to have a guarantee that okay now you reach this stage where you have maybe like a safe model or anything like this but it cannot be only by acting on the data set or empirical only because as we showed here you can always find a way to go in a new part of the space where nothing was seen before just because the space are gigantic right so you need to have a better parameterization or better control of
1:07:39
your network if you want to really have a provable guarantees for like rhf extrapolation in this case yeah yeah it's so interesting I mean and just to bring it home to folks as well I mean the example of of this jailbreak was I think you just go like a a a a you just have this like huge context length and then all of a um you've you've rendered The Prompt impervious of of rhf because rhf has a complexity ceiling right so like at some point you complexify the prompt you're
1:08:04
now outside the the you know the controllable kind of um space of of rlf so this is a big problem like what can we do to rhf to to fix this yeah yeah exactly and one thing to add is that this uh jailbreaking is not specific to one llm architector or one llm setting right so in the paper we showed some things with lama lama 2 but for example you know you experimented with chpt and you had the the same thing so it seems to be not specific to the architecture or the way rlf is implemented uh and so
1:08:35
there is really a fundamental problem which is a to control the behavior of a deep network uh in a really high dimensional space where you cannot visit all the uh places in this space so one way to look at it could be can we find better parameterization of the network so that only by learning from a few example we can probably generalize or extrapolate in many other parts of the space and this is about finding the right parameterization of the model or the right way to do our LF in this case
1:09:04
but this is a huge problem which relates to like extrapolation in general and just what to do with really high dimensional data and how to control the behavior of your model everywhere in the space only from a few uh training samples one uh way to attack the rlf is to increase the context length but one way to increase the uh success of the attack is actually to not just add the extra context with random tokens but to add related Concepts uh so because of the related Concepts in the added token
1:09:37
then the sparity of the attention will be uh less and therefore you have much higher chance to uh jailbreak the rhf mechanism so also those attacks are not easily detectable so for example if you just prep a a many times maybe it's easy to safeguard the model against that but if you just add natural English sentences with related concept to the toxic prompt you are giving it's really hard to detect right because it's a common English normal sentences but still this is able to jailbreak the RF
1:10:07
and even at a better rate than just with random tokens so there is also a lot of work to do in term of okay what is the relation between adding context that is related to the current prompt or just adding random context and all those things interact together and this is also a big future research question very interesting so um you've been working with um Yan Lon very very closely for many years at meta you're now at Brown what what is your research plan for the
1:10:33
next year uh yes so my research plan is to try to increase the amount of provable guarantees we have in current uh learning Solutions uh whether it's working with text or computer vision or like multimodal data set we need to dive deeper into all the things that are happening uh behind uh the curtains like training Dynamics uh Ness uh or biases that you learn from the data and reink the basic things that we've been doing forever like regularization or just like
1:11:04
uh training so that we can control for those things and give provable answers to uh users or practitioners and there are a lot of things to do because nowadays we don't question anything right and it turns out that we need to requestion most of those methods that we are using if we want to make progress in this area and of course this can take the uh shape of using spline but there are also many other tools we can use for this but the goal is that whenever your method is not working you
1:11:32
need to be able to give a precise answer and not just say okay try another hyper parameter and come back to me in two days so everything that we can do to have useful Theory uh provable guarantees uh and that is tractable to be applied to Industry scale problem this is uh what we're going to work on over the next few years very cool very cool and we were briefly talking earlier about about um you know the the DL Theory book with SH yider and Dan Roberts I mean do do you have any sort
1:11:59
of like broad views on on other um theories of deep learning well yeah so there are a lot and many of them have really practical insights right so for example you have this uh paper from Greg young where they show okay from theoretical characterization of training Dynamics uh and what happen in a current Network you can do cross validation on a small Network and then given the hyper parameters that you find uh you know it will you have a rule to extrapolate it uh so that it's also the best hper
1:12:28
parameter when you use a bigger model so you have a lot of really practical things like this that come from theoretical studies of De of deep networks I think right now what we are missing is something that is easily accessible for everyone and a lot of current theoretical studies of deep networks require a huge amount of mathematical background and it's really not easy to access for uh people uh that did not do like a PhD in math or at least a bachelor and Masters in math and
1:12:54
that's that's why I really like the spline formulation because you can actually make progress on it even just through visual inspection of the model so that's why I always try uh to not just do the for the but so that Anyone who reads the paper can learn something from it and train better models tomorrow so that's uh what I try to to focus on make it very accessible to people but yeah there a lot of different theoretical viewpoints of deep learning and each of them have been trying to
1:13:24
come up with new new solution uh but now I think we need to try to assemble everything into one that is uh independent of the modality independent of the architecture and accessible to to everyone as well amazing and are you going to publish a paper with Ellie pavick in the next year uh that's the hope that's the op yes amazing I I'm a huge fan of Ellie I had her on the show I think she's working a lot on negation at the moment in llm yes yeah yeah yeah and it's very interesting to uh talk to
1:13:50
her as well because the way she thinks about language uh and or we learn and olm learns and what we can uh try to learn from that is very interesting and really complimentary to try to use uh sir and splines to explain llms so that's why I think there will be some really fruitful fruitful collaborations with her in the next year for sure I mean in a way that's what's good about itml but it's what fascinated me about the papers that we talked about today because you've got this huge background
1:14:18
you know doing um that self-supervised learning and and vision models and and and so on and you know the the theme of of what we were talking about is this kind of spectrum of of complexity in in the representations and the training Dynamics and it's so interesting how you transferred that into rhf so it seems like a slightly different domain but like science is all about just reusing Knowledge from different domains in cross pollination right yeah yeah yeah exactly and I think the intuition you
1:14:42
get in one modality easily transfers to other modalities as long as you don't overfit too much to a really specific architecture a really specific Viewpoint and that's the beauty about splines because splines happen regardless of modality whatever insights you get about a spline partition is going to transfer the same across data set whether it's images or text Etc and that's all in this paper which is like a very good example as you said uh so you had the Sarat and Romain you came from the llm
1:15:12
expertise side and I came from the spline perspective and together it was very easy to come up with this solution because everything is transferable once you reach this intuition so that's why uh people should not be scared to uh explore New Dimensions new data modalities or even new architectures and that's actually how you get the the best inside that become complimentary at the end amazing now if people want to learn more about the spine Theory where would
1:15:38
you point them uh that's a good question so I think the best is probably to look at the few last papers that we've been doing with Rich imas uh that are about splines and there are a few different ones that are focusing for on generative model using spline to do uncertainty quantification or here for llms and try to find the one that is the closest to their current expertise so that there is not too many things to learn at once so only try to learn the spline partition
1:16:05
and so on uh and then to reach out to us obviously like feel free to send us emails and uh send us messages on Twitter and all uh because plines can be really cryptic if you look at uh papers from the 80s and 90s because the way they were thinking about it was very very different uh so don't try to look at a 30-y old paper what plan approxim o imation and approximation rat because it will confuse you more than anything first look at figures look at current
1:16:31
papers and then reach out if you have uh any questions cool and there's a bonus question so recently um kagor of Arnold networks came into the Limelight and they are also a kind of spline approach aren't they yes yeah yeah exactly so basically they try to present an alternative to current MLPs where they add hard code some uh spline activation function within the architecture and that's a really great example on how you can use a prior knowledge to really Define an architecture that is going to
1:16:59
work very well for some specific problems so I think most of the problems they were looking at were uh small scale and low dimensional uh and that's where you need really expert knowledge to design what the partition should look like and what type of spline to use to really get the best from small training set and splines also offer a really nice way to do this because again uh the geometry is visually interpretable you can see what you need and if you hard
1:17:25
code most of those things what it means implicitly that you need less training time less training samples to actually learn something meaningful so that's also another area where splines are useful is if you really an expert uh on your domain your data domain and your Downstream task you can then try to transfer this in term of geometric properties and spline give you a really nice uh way to do this and to actually put this in practice and and create new architectures and get new models R
1:17:53
bellri it's been an absolute honor and a pleasure to have you back on thank you so much likewise likewise very much thanks for the invitation
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat