Skip to main content
Loading Explore guides…
Language and Thought Development · Acedra
Home
Explore
Psychology
Language and Thought Development
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
183 segments
Copy
Hide video
0:02
all right thank you uh thanks thanks David for this amazing gift um Sally for your introductory marks wherever you are and Jim for set for being such an inspiring leader every time I see you talk about this I'm re-inspired all over again all right so my name is Josh tenam I'm one of the scientific directors uh I've been part of cbmm and the quest from the beginning and I'm I'm it's great pleasure and honor to be bringing you to the session here which is bringing you
0:28
the leaders of two of our missions um uh in addition to myself uh Laura Schultz and Rebecca Sachs are leading the development of intelligence Mission and Jacob Andreas and E federenko are leading the language Mission and um I'm going to do a bit of stage setting talk for about 10 or 15 minutes or so just to to give you a picture of why we're doing what we're doing what what what is the relationship between language and the development of thought which is on all
0:53
of our minds these days and then we're going to have a conversation a structured conversation where you're going to see the many kinds of things that this team is thinking about how we're thinking together how we've been working together what our next steps are and some of the kind of impact on society that uh Jim talked about but let's start off with our North Stars where do we hope to get to well imagine if we could build a machine that grows into intelligence the way a person
1:21
does that starts like a baby and learns like a child this has been the oldest dream the original Vision going back to Alan Turing for how to build an AI system and for good reason because if you think about it a human child is the only learning system in the known universe that reliably robustly reproducibly grows into full human intelligence starting from less so if we could understand that it would be a road map a guide maybe its own kind of scaling route for building AI but of
1:53
course more fundamentally it would be a way to understand ourselves where our own minds come from if we had something like this think about all the other kinds of Technology this could enable imagine if we had robots that you could trust to help you out around the house and learn new tasks and new environments as quickly and as easily as another person does or imagine if we had systems like chat GPT which you could ask any question and expect a reasonable answer but inside they're not just black
2:23
boxes trained on all of Humanity's data instead there are best current models of how human minds think and communicate these may seem like science fiction but increasingly they are starting to seem kind of attainable and the the research that's going on in the quest and all the ways in which we're also working with outside groups are quite seriously um I think taking steps towards delivering what we hope will be the positive impact versions of these ideas but AI today
2:51
let's look at AI today a little bit right I mean I shouldn't say but AI today and AI today AI today is remarkable it's you know it's gone from an academic field to a thing in everybody's lives and affecting every part of our society and there have been remarkable systems real in some sense prototypes of these things robots that actually are there in the world making life or death decisions otherwise known as self-driving cars like Tesla or weo and they save lives and they take lives
3:18
and so far they I think they save more lives than they take lives but it's unclear um or language-based AI conversational system chat GPT or Google's Gemini which is now you know you can't do a Google search without having an AI chatbot giving you answers whether you like them or not right um so we're going to start off in the session reflecting on what are some of the similarities and differences between today's AI that these the biggest most valuable companies and the world's
3:45
richest people um have been leading and batting on what are the similarities and differences between these systems and what we've come to understand about our own minds and brains and all the ways in which both the similarities and differences are opportunities for these bidirectional interchanges between the science and engineering of AI as Jim talked about we'll talk some about what are the activities we've been doing to address these opportunities where are we
4:07
going next over the next two years and Beyond and again what are some of the impact so I'll give you my perspective first um again these are amazing systems it's truly remarkable that we that we actually you know have these robots in the world okay and yet there's something sometimes head-scratchingly weird about where the successes and failures how they pattern together so for example weo can you know drives truly autonomously but only in some parts of a couple of
4:38
cities Tesla you can drive on any Road in the United States but you might have to intervene or you do have to intervene on average every 13 miles so there's something very different here and we're we're far from having a complete solution even to the limited version of robots that just stay on the streets okay despite hundreds of millions of dollars of investment a lot of product is being made but it's it's it's hard to say how long it's going to take to get there or like in conversational AI
5:03
systems so Gemini this is Google's version of chat GPT um this is the latest most advanced model okay you can you can ask it I can show it the picture I just showed you and say tell me what's in this image and it will do a great job of describing it if you can't read it it says the image shows the interior of a Tesla vehicle specifically the driver seat area with the dashboard all this stuff the accompanying text is a headline um and subheading from an RS Technic article discussing the
5:29
limitations of Tesla's full Sur driving system amazing okay and it didn't just get that because that's actually didn't just get that from the web that's a mashup I made that's not an actual web page but it did a good job of seeing it looks like an actual web page but suppose we give it something that's a little bit as we say out of the training distribution a little bit less of a typical image this is one from my colleague vikash man Singa who you've heard already a little about you'll hear
5:51
a lot more from him later today UMES this is a image that's designed to test a um uh you know probabilistic inference Sy systems of the sort you'll see here okay um you can all read it what does it say uncertainty um how confident are you how certain are you any uncertainty no you you know what it is okay but this very same system says the image shows the capture 52g csih 5 zb2 well it recognizes it as some kind of a capture that's reasonable but that's that's not the right answer and
6:22
when asked how confident it says well I'm around 70 to 80% confident the GL and two I can see although they're the ones I'm the least certain about but the rest I can see pretty relatively clear you know what's going on with that or maybe maybe it's just a little too hard so we can make it a little bit easier um now uh we just showed it certainty what is this show the image appears to be a very low resolution and noisy rendition of the word capture it simulates the
6:46
visual challenge aspect of a capture test huh how confident are you again I'm 75 to 80% while the image is very noisy the general shapes of the letters are discernable the PT and the second C are the clearest okay so what's going on well again to to to uh build on some of the themes you heard from Jim what's driving these systems again amazing technology that's designed to simulate what in some sense um the input output relationships of a mind to take the inputs that we might have with
7:18
our brains in the world and produce the outputs by training what is effectively and in some important way just a big black box full of very simple little elements that when you scale them up and make them big enough and train all the parameters on all the data you can measure or simulate from the world the internals of this blackbox system in in some ways really quite surprisingly have these emerging properties where something really genuinely intelligent seems to emerge and yet it's
7:43
unpredictable and clearly very different than the way our minds work okay and we'll we'll again dig into that in different ways um this the what you sometimes heard of the scaling laws this is where some of the interesting theory of AI comes in whether it's these scaling laws of learning or the scaling laws of inference which we'll be talking about later okay inference is more like online thinking scaling laws of learning is what happens when you add in more and
8:05
more data and offline comput the the the the millions and increasingly close to billions of dollars that are used to train these systems some performance reliably increases every 10x compute is another increment of performance but it's performance on the things that you can easily measure if you're a big tech company which is like which is you know how well you predict the next letter or the next thumbs up or maybe the next dollar but not the next increment of
8:31
intelligence nobody yet knows what are the real scaling laws of intelligence so it's these are very unpredictable bets in contrast something's something different is going on inside our minds our minds are built as Jim said to model the world that is the physical world and also the social world so I'm going to turn it over now to my colleagues starting with the developmental researchers um to to to say something about what is what is different about the way our minds are built from the beginning
8:59
so go ahead Laur yeah terrific um uh first of all thank you all for being here and it's a real pleasure to be part of this incredibly inspiring and ambitious uh event um I think you know we all know that today's industry AI is actually getting closer and closer to humanlike Performance and we also I think all know that today's industry AI is deeply and really unlike human uh intelligence with respect to learning and the the simplest respect in which we all know that is true is the amount of
9:33
data that it takes um so children learn from tiny amounts of data sometimes just one or two examples um we can't learn from Big Data industry AI can but it has to it doesn't have any alternative choice and to give you a a comparison I think the amount of data used to train gp4 relative to the input to a human child is approximate the difference um between the population in this room and the population of Tokyo so that's a very uh very different scale of measure um
10:12
that's that's the thing I think we all know but I think there are at least three other respects in which Children's Learning is really different uh human learning that is to say from uh AI learning uh uh an artificial neural network will learn from whatever you give it it will learn from any data set at all for better human children do not do that um babies uh are selective Learners from the start they choose what to look at they choose how long to look at it they choose when to look it away
10:39
um and as soon as they can act on the world um they pursue their own goals their own problems nobody in the history of the universe has ever told a toddler to try to put clothes spins on a cat or Play-Doh h under the couch to catch a Voss Raptor um uh children do things like that every day I did not I promise you tell my daughter that what she should do in the library was climb the shelves of the children's room she decided to that I promise I knew they were bolted in before I snaap the photo
11:09
um but children children choose what they want to learn and they explore in that way and they don't just learn from data they learn from other human beings um they ask questions dozens of times a day and the questions they ask are ad hoc idiosyncratic often maybe the first time that question question has ever been asked in the universe they invent things to ask about and the information they get and the information they share at 10 months they start pointing and
11:38
showing us what they want to know um they take your hand they take you where they want to go um the information they get the information we share they share and we share with them is neither arbitrary um nor is it a representative sample of all observations it is tailored specifically to their abilities their knowledge their interest and their goals and I think something we can sometimes forget as cognitive scientists and and as AI Engineers is of course children don't just learn and teach from
12:08
other people they want to connect with other people they care about them and they care for them and this is a really important and central part of what it means to be human what our motivational system is like but I think maybe the least recognized uh and maybe most fundamental way that human learning differs from today to's AI is today's AI is profoundly not developmental in fact in some sense it is profoundly not about learning because think how odd it is
12:41
that we have systems that can do better than human biologists at protein folding and they don't know what a protein is and they don't know what a cell is and they don't know what a body is today's AI can start very very dumb and become very very smart and that's nothing like human learning humans are intelligent from the start we start out intelligent at simple things and then we become intelligent at more and more complex things our learning is cumulative and I think the Grand
13:11
Challenge for the Next Generation AI is to understand and engineer an intelligence like this and intelligence that always intelligent and where it's cannot and does not have radical gaps and discontinuities and it's learning because because knowledge grows
13:34
systematically so fun thanks um yeah so I'll just continue on that theme and talk a little bit about the work that we've done um actually on exactly the same ideas that Laura just mentioned but now with babies so Laura is talking about the selection of input from a 10-month-old this is a four-month-old actually each data point that each point is the point of gaze of a four-month-old um preverbal baby who could look at anything but you see they choose what they're looking at really
14:01
systematically all the babies are looking at the faces and the hands so these are babies who before they can do move any other part of their body are choosing how to learn from the World by choosing what to look at and one of the Privileges of working at MIT is that we can study that initial architecture the architecture that drives what babies choose to look at and learn from from before they can move um and we can do that using neuroimaging so Josh was talking about the um model of world
14:29
inside the mind we can look at it literally the model of the world inside the baby's brains and one of the places that we started this research was looking for how people see faces and each other this kind of core motivational system we've known based on incredibly important discoveries at MIT I don't know if you all know that Nancy kisher won the Cav prize the highest prize in Neuroscience for the discovery of these face regions in human adults and we've been continuing that work
14:55
trying to ask where do those come from right do we have to learn them slowly with hundreds of thousands of hours of experience like an AI system and of course we don't um so we've been developing systems for brain Imaging of human babies starting from two months and what we've discovered over the last couple of years is that the system for seeing faces that adults have is all the whole architecture is there by two months old the earliest we can measure it a really powerful example of the
15:23
thing Laura was telling you that baby brains start with all the tools to learn from the world that they're going to experience um so let's now turn over to more the language side yeah starting with EV sure um hi guys how do I click to advance sorry the right button yeah okay okay so um uh I spent a lot of time uh in my in the last like 20 years thinking about um human cognition and how language might figure into this so with these kinds of early developing capacities that Laura and Rebecca
15:57
mentioned by the time we are an adult we can do all sorts of things in the world we learn Concepts from very concrete ones like what an appolis two very abstract ones what is Justice right we learn to reason about Concepts and the relationships between them we learn to um understand the physical world how gravity Works how objects interact but also the social world how agents relate to each other how to cooperate to achieve goals we also learn to reason in fundamentally abstract ract ways that
16:29
are not really tied to any particular knowledge base we can solve novel problems on stimuli that you've never experienced before and this kind of uh ability to reason abstractly is tied to these very fundamental capacities like attention working memory understanding what a goal is and how to work toward a goal um we create and appreciate art we can estimate numerosities we invent strategy games like chess um and uh the question that has been puzzling me for
17:01
many years is how does language fit into this to what extent is our cognitive sophistication enabled or unlocked by language and um of course A lot of these abilities in some form we share with other animals I'm marking those with green check marks right in some form they exist in other um in other species but there's also some which I'm marking in red that seem to be specific to humans and in some ways some of these human specific abilities are in some sense trivially enabled by language
17:33
through the accumulation of knowledge that language enables us to pass from one generation to the next if we had to reinvent things from scratch in each generation we couldn't figure out calculus we couldn't build and program machines we couldn't create um narratives that capture the history of the humankind um but when I started working on these questions I was interested in a bigger claim which is the claim that we use linguistic representations words and structures to
18:03
actually engage in thinking which is a claim that many people have made over the years so these are just example quotes from chsky where he wrote things like the systems of thought use linguistic expressions for reasoning interpretation organizing action and other mental acts and he further speculated that a substantial part of what we call thinking is simply linguistic manipulation so if there is a severe deficit of language there will be a severe deficit of thought and he kind
18:30
of States these things as if there are truths facts about the world but of course without data they're just hypothesis and I wanted to test them empirically and so over the last bunch of years with a uh many different colleagues we've used that two approaches to try to ask whether these um ideas are actually on the right track and the way we can ask about the relationship between language and other things is in two ways one we can find your language areas if you come into the
18:57
lab and we put you in the scin in about 5 minutes we can find the parts of your brain that do language and we know that reliably and then we can ask how are these areas working when I'm asking you to solve a math problem or I'm asking you to read a piece of computer code if you're a programmer or to solve a logic puzzle like Sudoku right and we find time and again across many many different tasks that your language regions are actually silent when you're engaging in many many different kinds of
19:22
thinking other parts of the brain are working hard but the language representations don't seem to be engaged and in a complimentary approach we can look at patients who have problems with language um Aphasia right so some of you may have relatives who have this condition typically as a result of a stroke you um may lose some parts so sometimes the entire language system and lose your ability to produce and understand language and again we can ask okay these patients have trouble
19:47
communicating linguistically but can they still think like if they could play chess before can they still engage in chess if they were writing music could they write music if they were a mathematician could they still do math and the trans that a lot of these abilities appear to be preserved so it doesn't seem like even losing language can still allow you to engage in these very sophisticated mental acts so the answers to these questions seem to be that no the language system is not
20:13
working when we're doing these kinds of um thinking tasks we don't need linguistic representations per se um but of course language is very deeply embedded in everything we do in the world like it's very hard to imagine our everyday life without language and I think next um Jacob will talk about the implications of this dissociation between language and thinking for how we build better machines let me just do one um sort of again stage setting things so you've heard now from three of my
20:40
colleagues in brain and cognitive science like laying out from their different areas of expertise like what we tend to what we've come to understand about human minds and and the relationship between language like the ways were built for intelligence from the start the ways language adds on to that in clearly amazing ways it's clearly our a superpower of humans right as I was saying I mean I mean it trivially enables culture but it enables culture and it it's it's the it's the
21:03
most obvious difference of us from from other animals um and yet we see something that might look sort of different from the route that we're seeing with today's amazing language models um following a different path so Jacob is is a member of Cale he's one of the world's experts in natural language processing and language models of all kinds and so let's turn over to Jacob to say like tell us about some of the differences uh there yeah or similarities and differences thanks yeah
21:27
and I guess to start by sort of setting the stage steer I want to say something that basically everybody on this stage has already said in in uh computational terms that when we think about uh you know what it is that's going on in human learning all the things that Laura was talking about you know such that when you're born you're able to immediately start sort of engaging in uh complicated exploration and play and and learning from very small amounts of data um
21:52
obviously these things don't come from nowhere they are the result of a very long evolutionary process that you know where Evolution even if no individual learner has seen you know data comparable to a learner's data like the the room scales with Tokyo um but I think an important uh difference there or an important thing to keep in mind is that uh almost none of that data to a first approximation none of that data is linguistic right whatever it is that Evolution has seen whatever it is that
22:21
Evolution has done that puts you in that magical state that you're in uh as an infant when you start learning uh really has very little to do with uh language specifically it has everything to do with you know the kind of more General families of reasoning and learning problems that you know the sort of entire history of life has experienced um and this is totally different again from the ways in which we're training language models right now and you know really sort of AI systems of all kind
22:45
right now uh where the very first thing that we do is we download all of the text on the internet the sort of end result of uh that process that has generated humans in their current state uh and we train a really good predictive model uh on that text on that language data on all of these sort of uh you know sort of accumulated knowledge accumulated uh linguistic data uh that are exactly the things that human language Learners don't need that much of um and so when we start to think
23:12
about what it would look like to build more human-like intelligence uh and what it would look like especially to build machines that can kind of learn on the same time scales from the same kinds of data as humans do I think this is really the sort of big difference that we have to Grapple with and to think about what are the kinds of things that we could build in or at least what are the kinds of other learning problems and the kinds of other uh data we could present our
23:33
learning algorithms with uh such that they would really be able to make do with very little language data and generalize from very little language data but also all of the other kinds of experience that uh that human children have access to um without having to to do these large scale things and I think this would you know obviously have sort of engineering consequences in terms of the amount of data that we would then need to train models and uh the the amount of work that we would need to do
23:56
to find- tune them but also uh really I think fundamentally change the kinds of generalizations that we would expect to see from these systems the kinds of robustness uh we would uh expect to see from these systems and you know I think if we're going to answer this question then we also need to be able to start to ask even more basic questions about uh what kinds of non-linguistic capabilities what kinds of general purpose reasoning abilities what kinds of you know sorts of algorithmic
24:20
thinking and algorithmic generalization might we expect to see in uh systems that look like modern language models so that we can start to think about what those other data sources and what those other sort of architectures of the future might look like and so one of the things that we have started to think about a lot here um is exactly these kinds of basic questions about can you in this function fitting Paradigm that Jim and Josh were describing before um even uh Implement anything that looks
24:47
like algorithmic reasoning and what can we say about the kinds of algorithmic constraints uh that exist on the sorts of models that we have right now and you know I think what we're seeing so far is that this is actually a fairly complicated picture on hand with exactly the right data and with exactly the right amounts of the right kind of data you can actually take a thing that looks like a modern Transformer language model uh and get it to implement you know sort
25:08
of inside its head so to speak things that look like classical learning algorithms things that maybe look a little bit more like what it is that we might want a humanlike language learn to do with small amounts of data um but incredibly inefficiently and uh you know with incredibly sensitively to the kinds of data that uh that they're trained on um and you know maybe bring this more explicitly to the point about uh World models and the extent to which we should
25:32
expect things that look like modern language models to actually build representations of the outside world uh to plan to reason to simulate to do things like that internally you know another question that you can ask and something that I think people are talking about quite a lot in uh in the sort of Industry AI world the world of language models and and and other kinds of large scale machine learning models is the extent to which these big neural sequence models contain uh things that
25:56
look like World models or whether you can learn things that that we would be you know even with our Neuroscience hats on uh inclined to call World models uh just as a result of training on very large amounts of language data um and again the answer here is messy but I think uh mostly that you get only very rudimentary forms of these models and only from very very large amounts of language data and I think fundamentally this is because language data is just
26:18
not the right kind of data from which uh we should really expect these models to emerge um and so we'll talk a little bit more about I think some of the sort of concrete work that we've been doing in the scope of the quest uh to address some of these limitations and to think about what other learning algorithms other martal architectures other data sources might look like but you know maybe the main point that I want to make now is just that um on one hand I think
26:42
there is starting to be promising evidence that you can get some kinds of structured representations inside these big neural models if you set them up in the right way but on the other hand the ways in which we're setting them up right now really seem not to be the ones uh that even you know either if you want to build sort of like an explicit computational executable model or if you want to do something more implicit that lives inside one of these neural sequence models that you're not going to
27:04
get it just by training on kind of all the language data in the world and nothing else um oh uh and there's one more slide which I guess is going to sort of queue up the next section which is that uh you know one thing that you might ask then uh is what other paths we have and what other kinds of models uh might look like that could do this um and I think one thing that we're actually going to spend quite a lot of time talking about today is that you know obviously there's an
27:26
enormous tradition of work much of it actually coming from people who are on the stage right now at MIT at building sort of explicit uh and explicitly computational models of how human intelligence works that bottom out you know not in sort of Transformers and generic neural architectures but like real computer programs symbolic descriptions of the kinds of computation that people are uh performing and one thing that I think is really quite exciting about uh these kinds of models
27:53
right now is that as these more general purpose uh neural sequence models get better and better they actually give us also the ability to build better more explicit symbolic World models and I think this is actually a sort of place that we see um the field moving as a whole but towards large scale neural sequence models that are reasoning about the world that are simulating the world uh not just by doing this kind of implicit neural computation but by actually building uh and reasoning about
28:20
and executing explicit computational models of of how the world Works um and with that I think we're into the next I could we have two different control okay yeah no I'll just do it from my left so so I just want to sort of again sum up the picture that we've seen and then talk about like this sort of the background of of what we're doing which I think is can be summarized with by with these two sets of bullet points so just as Jim said like if you're going to
28:43
if you're going to remember anything from from this session I think this is a set of important points which I'm trying to summarize in three bullet points each like what is the essence of the kind of root to building intelligence that we're seeing driven by today's massive investments from industry and on the on the other side what do we see the way intelligence seems to grow in human minds and brains and what might be another bet another scaling path perhaps
29:08
for tomorrow's AI right the quest and the quest researchers are tools the our our virtuous integration between science and engineering can address both of these they can help us better understand the left side and better build the right side okay but just to sum up these points right in the sense the the industry scaling path the Deep learning massive data massive compute has a notion of learning in quotes that is um or that's the optimization path Jacob is talking about which is simple and dumb
29:36
by Design in order to be scalable right the the kind of learning that goes on in the weights of a neural network isn't supposed to be smart it's supposed to be just a small little gradient step on some big endtoend differentiable function okay but the idea is you do that enough and something that you could maybe legitimately call intelligence in some form is emerges as the end state right it's surprising remarkable but also kind of fragile because intelligence and the Hallmark of
30:02
intelligence is the ability to solve new problems or generalize generalization is another one of these emergent properties it's not built in but it depends on similarity to the training data okay um and and as a result it's unpredictable and again like like I showed you in the examples we saw some kinds of generalization are predictable but the ones that are really about understanding the world are unpredictable and then crucially the relation between language
30:25
and thought the reason why we're up here so much focused on language and the the reason why the big transformers the big machine learning models the only reason why the entire world is now talking about and using AI in the way that they didn't before is because language is a superpower and training on all the world's language data that's ever been produced for Tokyo City size lifetimes 100 you know um is something but again it's this idea that that intelligence
30:51
has to or that that that intelligence emerges or thinking emerges after language distilled out in some or backed out from language in contrast to the human route right where we're built to be intelligent from the smart it's intelligence in some form it's not that of course adults are are in in many ways more intelligent than than young babies maybe not in all ways but intelligence changes with learning but intelligence is crucially a thing that's built in from the start that lets you learn so
31:18
much from so little that's fundamentally oriented around our our brains and Minds being built to have certain kinds of models of the physical world and of each other of each other's as agents even with even even faces and and the interface through our eyes and how and our shared attention and then this idea that thinking is independent from exists before language and language grounds on that so what what we're doing especially in the developmental roote of the quest
31:42
this is where this I'm just now going to tell you a little about my research is we're building computational models engineering style models designed to reverse engineer and follow this alternative scaling path so that means understanding what's built in and how do you learn the rest that's this developmental road map and and again to to take the kind of things that my colleagues have been talking about here and that we in our collaborations build together it's understanding the
32:05
representations and algorithms so not just function approximators and optimization but symbolic models of the physical and social World probabalistic models fundamentally the kind of intelligence that that we that we see not just in humans but in all in all brains as EV showed you can think of as a certain kind of rationality making good guesses and Bets about what what's going to happen next what are the effects of your actions what might happen if you if you acted in a
32:32
different way it's that ability to to predict and to even imagine ways the world could be and use those to plan your actions that's what brains do from the beginning well before there were humans in the world brains were doing this kind of thing in humans those build fundamentally on these models as we've been uh fleshing out here we'll we'll see more about this models of the physical world but especially the social World other agents goals and understanding how agents actions depend
32:59
on what goes on inside their minds our models of other Minds their goals and their beliefs okay and especially this Focus as you heard from Laura and Rebecca on goals on understanding even the youngest babies have goals and how they look and how they engage how they learn and how they connect to others and how that and learning about the goals of others is so crucial to how our intelligence develops so the developmental road map and again if you want take one other thing it's sort of
33:24
this part here it's what you know what is built in in terms of this core architecture and then if you want you know what we best understand in very high level terms about the stages of human cognitive development you can take these three there's basically before language language and then after language what is the what kind of learning goes on within the core architecture um for the first say year and a half of life then the major focus of learning over say one and a half to
33:49
age three is learning language and then once you have language how do you use that to learn everything else all the knowledge that accumulated culturally right and that is that you that you get socially from others and in order to to engineer this to build models and test them this this can't rest just on the industry Computing stack just bigger and bigger Transformers you'll hear more about this in the scaling inference mission in the afternoon but we're building and this is vikash man Singa
34:13
group and probabilistic Computing group at MIT is building a new Computing stack you'll hear more about probabilistic programs but new kinds of programming languages that integrate function approximators like neural networks but also symbolic knowledge representation symbol olic languages programming languages for expressing all the knowledge of how the world works and what you can do and probabilistic inference on top of those inside and on top of those to make good guesses and
34:37
good bets not just about where a ball might bounce but what somebody's thinking what what we might want to do together okay so that's so that's that's what we're doing up here um now I'm going to turn it back to again my developmental colleagues in a second but I'll just say what we've been doing over the first two years of this phase of the quest is trying to trying to build trying to use this tool to capture for example some of the things you saw about and you saw this in Jim's talk here
35:02
where like the most basic ways that even from infants who are Oney old or less can understand what other agents are doing even from kind of weird stimuli like so experiments with 10-month old 12-month old babies show them things like this you see a blue and a Red Ball moving around but you don't just see them as balls rolling on a table you see one thing is chasing another and the other one is trying to get away or here you see a a red ball that appears to jump over a wall to get to a blue
35:33
triangle and in experiments from our our former uh Quest colleague now um postto in Rebecca's lab also worked with Laura and me Sherry Lou she's now uh faculty at John's Hopkins but continues to work with us as part of the multi-university efforts of the quest study for example how infants make understand how much an agent like the red one wants to be with someone based on how much much work they're willing to do to get to them if even 10-month old babies understand that
36:02
if you're willing to jump over a higher wall there's a lot of songs written about this you must really want to be with the blue one okay so um whether it's that kind of thing that's even present in infants or in classic work with one and a half year olds these are from the studies of warakin and Thomas ask what lets that one and a half-year-old see an adult doing a weird action that they've never seen before and just the first time get a sense of what they're doing and how to help them
36:28
right what is the intelligence that is you know there so early we're not even talking toky I mean this room size we're talking like yeah okay so the the kind of models that that we've been developing to do this involve you're you're sort of seeing a sketch of it here this this is not a model of of your brain it's a model of how your brain models other brains as Rebecca has long said that's a good way to think of a theory of Mind a probabilistic causal model that lets you observe action
36:58
as the effects of some agent planning you don't see somebody's beliefs or goals but you have a model of how they take their goals and beliefs to plan actions that they might reasonably think are efficient that in some form mathematically can lead to achieving High reward or G or or rewarding States subtracting the cost so this is this is what our our again former Quest student Julian har Edinger who's now an assistant professor uh at Yale hopefully not for long um assistant professor that is
37:29
uh the um uh he uh um you know what he called the naive utility calculus but this idea that you know thinking about maximizing expected utility is something that in some form even 10-month-old babies do to try to analyze what goes on in these these studies and by building computational models indeed simple kinds of probalistic programs of these sorts of things um of these sorts of abilities that you can see in young infants we um this is work that was in Vash man sing's
37:58
group with Shen tanii Shen who's a another one of our great fantastic Advanced PhD students um we we we recently joined a competition actually that DARPA the defense Advanced research projects agency ran in trying to build machines with this infant stage of Common Sense and in the theory of Mind part we just killed it so the basian theory of mind you can see we got like 98% on all the tasks that were set up these simple kinds of infant benchmarks but we're not proud of that we're proud
38:27
that that our systems you know which built in the kind of thing that we see built into young babies do in some sense very well on a couple of AI style benchmarks inspired by developmental psychology but these don't solve the whole problem we still don't have robots that can help you out on the house and learn new tasks and fundamentally again going back to what Jim said we need a much better bigger faster more virtuous cycle of experimenting in and and benchmarks to drive our algorithms these
38:56
very simple first steps need to go much um more and faster so let me turn it back to our developmental psychologists who are working on this FL um so I think you know without doubt the fundamental bottleneck in um getting data about human development has been uh recruiting participants so up until a few years ago every single data point to a first approximation that we had about any human child involved calling a parent scheduling the when they were free next Thursday at 10: and if the
39:30
child wasn't sick and the parent remembered they got in a car strapped the child into a car seat found a parking space somewhere pulled the child out of the car seat and if the child wasn't asleep or fussy and was willing to come into the lab uh then you got a data point and so every single data point was incredibly hard one and research was slow and if you wanted to go back to that same child again and test them repeatedly that was essentially heroic research um thanks to a h brilliant graduate
40:01
student in my lab Kim Scott in 2015 we started building a platform for online automated developmental testing in 2015 we didn't know what was coming and one of the things we were worried about was whether we could get other researchers to buy into this and to sign up because of course if you want to drive parents to a site on the internet you need content and you need to refresh that content and needs to be new all the time and we were planning to launch this in September
40:33
2020 and then the world shut down in March and we were able to open this site a few months earlier and uh quite amazingly I have my paper here so I get the facts for uh today we have 262 Labs up on children helping science from 1676 different instit tions uh around the globe there are over 1300 scientists active on the site right now from 14 different countries since the site launch 13,000 children have participated unique children in studies and almost 15,000 are on the site right
41:14
now ranging from newborns and adolescents and uh up until now the site has been uh you you you look for this in your uh search engine you can just Google you know developmental studies and children and find this and register online and despite the ongoing digital little divide one of the things I think is quite uh important is that this site is recruiting a population that is closer to US Census Data than almost any data set ever collected in developmental
41:45
Labs see above on how we used to have to recruit parents to come into universities um uh uh right now uh 18% of our population on children helping science has only a high school degree um so this is a really unique population and as of this fall um we are about to and thanks to the heroic work of which is also heroic of Melissa Klein spru who I think is somewhere here and who is the current executive uh director of children helping science uh we're about to launch this on phones and tablets as
42:19
well so I think we can really expand the reach why does this matter I well I think it's not overstating the case to say this has been a game Cher in the field we can now get data overnight we can these are by the way all studies that are up there right now the second running currently um uh we can get data overnight now on children you can do longitudinal studies now easily which means we can test the effective interventions we can do pre- and post testing in a way that is no longer uh uh
42:51
horribly uh uh time consuming and expensive um and really critically we can get the kind of data you need for AI benchmarks and the kind of data you need uh the Big Data you might need to test graded effects in order to um uh test the predictions of these computational models so it it's it's a new platform I think we're just starting to tap into its potential but I think you know a decade from now especially together with the tools I think uh rebec is going to
43:21
talk more about for automated coding um uh the picture of what we know and can learn about human Learning and Development is is going to change uh massively thanks yeah so I'll just give one quick example of something we were able to do as a result of this platform and the um teamwork of the quest um so like Laura said for um developmental research it used to take us approximately two hours of time for each individual data point from a single infant um the in the most recent project
43:55
um supported by the quest over the last couple of years we were able to replace each of these with a um automated process so um look it and children helping science to recruit the families instead of having an experiment or have to be there for the appointment and um kind of code the experiment as it was going live we implemented automated um parent training so parents can run the experiments themselves from home at any time of their choice um and then thanks to amazing advances in models of
44:23
perception the kinds of models actually that Jim was talking about we can replace a um an MIT undergraduate coding every frame for where the baby's looking uh with an automatic model of that codes the the baby's G um gaze automatically by um using computer vision this means that it takes the experimental about five minutes to check the data from every infant instead of two hours and this lets us scale up research in all kinds of ways I'll just give one one
44:49
little example of where we got started I told you infants choose what to look at um and this is an example data set for a very simple phenomenon in babies which is in the blue bars on the left under the babies what you can see is how long a baby looks at a um a little animated animal depending on how many times the baby seen it before and the um red line is for a new animal this is a incredibly classic phenomenon about baby looking that they get less interested in
45:16
something they already know right so you can see if they've seen it more times before they will look less and less that's the blue line going down um what we were able to do is um take the exact images that the babies had seen all these little animated uh animals and vegetables and run them through Jim's model his CNN model of vision to tell us how similar each object is to each other object and we use that with a probabilistic program like the ones that uh Vash makes to model rational
45:41
exploration how much would you look at the next object if you were trying to rationally explore the world and make a rational choice between how much you expect to learn from this next image versus anything else you could be doing with your eyes what I'm showing here is the prediction of that model without fitting this model just predicts that um exactly what we saw in the babies that you will look less and less at The Familiar objects um you will look continuously you will continue to
46:08
sustain attention on novel objects and not shown here is we could then generalize this to stimulate that this model had never seen at all like what if instead of one cow now there's two cows right and the model can likewise make predictions completely out of sample so this notion of being able to generalize Way Beyond into the Corner cases Way Beyond the examples um in the data and the other thing is that this was possible at the scale of 144 babies in this one little experiment because of
46:36
look it right if we had been trying to get 144 babies the way I did this kind of research when I was trained you know that would take four people's phds to get this one experiment done and instead this is one piece of one PhD now and that's at the beta version that's just as we're getting started so yeah so now let's um keep going on this theme of like things we been doing especially some of our most exciting integrative Quest projects in the language mission that that connect
47:02
both to measurement modeling as well as development so yeah go ahead Jacob and and that yeah yeah thanks so and coming back actually to some of the things that Jim was saying in the beginning of this a big part of you know making scientific progress is just being able to sort of measure progress in the first place and when we start to talk about building uh computational models of of these more sort of General cognitive phenomena and I guess especially about uh uh or
47:27
especially models of uh of the outside world and measuring AI systems and natural systems ability to to build models of the outside world um a big challenge here especially in the I think machine learning world has been that we just really haven't had good data and good evaluation metrics for these things in the first place you know almost all of the kinds of evaluation benchmarks that people um use for evaluating modern AI systems are collected much in the same way that the training data is you
47:55
just download whatever you can from the internet you don't think too too closely about the sort of distribution over uh examples and and problem instances that you're asking people to solve um and so one of the like big sort of quest wide projects that uh we've been involved in that was led by uh former Quest postto Anna Ivanova who's now a professor at Georgia Tech um was just to build uh like really detailed really comp comprehensive Benchmark of uh sort of
48:20
General World modeling abilities really grounded in what we know about the cognitive science of these questions um and so have coverage of a bunch of different domains of reasoning uh in the sort of social domains quantitative domains physical domains um and you know one of the things that and you know and so we we we have built this we've released this uh it's in the process of actually being integrated into uh the Stanford Helm thing which is like the big major uh benchmarking system that
48:45
all the big tech companies use for for evaluating their models um and the surprising thing here I was sort of expecting that you know with a lot of these simui we would plug them into state-of-the-art models and they would be simple enough that uh you know the the very best Frontier you know sort of open AI Google kinds of things uh would already do quite well at this Benchmark and it would really just be useful as a tool for uh understanding what's going on inside these models when they're
49:08
solving these problems and in fact that was not the case is that what we see really across the board is that even with state-of-the-art models right now um systems out there are just very bad uh at uh a lot of what you think of is very basic social reasoning very basic physical reasoning uh if you evaluate it in a sort of sufficiently Den sufficiently rigorous sufficiently quantitative way um so this has been one of the exciting things is that now we actually have some measure of progress
49:32
that we can use as we start to build the sort of next generation of models to to figure out how we're doing um you know and so what does that look like what does that next generation of models look like um to come back to one of the things that I was saying at the beginning about the differences between the way we train modern AI systems which are sort of language first and then maybe a little bit of everything else uh and the way we think about human development um what would it actually
49:58
look like to uh to invert this equation and to to sort of build training procedures and training paradigms that can make use of Rich grounded embodied supervision of the kind that human Learners get like really right from the very beginning of training um and so this was an effort that uh chunguang who was another uh joint quest postto in in Evan my labs um spent a bunch of time working at um and surprisingly found both that the sort of modern recipe of do a bunch of pre-training uh and then
50:26
try to bolt the and stuff on at the end really doesn't work quite well if what you're trying to measure is uh even sort of knowledge of language uh as it relates to the world and being able to generate and reason about language that's talking about sort of physical embodied things um but what he was then able to do was to sort of come up with new new model architectures new ways of training these models that for the first time ever actually allowed you to get a
50:48
language model uh even a system that was just better at generating Language by virtue of training on a bunch of vision data and not just on more language data okay can I just say one thing as one of the quest leaders and fan of this work but not participant um you know I think and this goes you know what Jim said or some of his worries are also some of my worries right it's like part of in the quest like what one of the things that keeps us up at night as research leaders
51:12
is how do we make a new kind of science where teams can do things that people couldn't do before so these two projects you just saw the first one that Ana avanova LED that was that was a a massive team by academic standards there almost 20 researchers mostly grad students and some postdocs but four five pis and it was you know it was our first Quest seed funds that made that happen this was a small team just these two plus changu I think um but again it was part of our first Quest seed fund for
51:36
the language Mission and they did this great thing that won a best paper word I'm allowed to brag about it because it wasn't my work um this is one of the m one of the top conferences in computational linguistics only two papers one best paper is this was one of them for this last year and Not only was that but the citation for the paper said it was especially like f fundamental and innovative and especially the cognitive the cognitive perspective was especially
52:00
Innovative on these basic things so it's in a in a small way but I think it shows that you know again kudos to you guys and it shows that the the bets we've been making our first steps I think they they really can pay off and and move the needle in the community Y and I think um in general uh trying to leverage these models to understand development is incredibly exciting because one really hard thing about studying development is we can't easily manip ulate things for
52:27
good ethical reasons right we can't mess with kids inputs and see what happens right so typically a lot of different kinds of things come together during these first few years and many things co-develop at the same time but using these models we can try to isolate different pieces and try to understand how they may affect language learning and then subsequently learning about the world through language so in addition to Chang xu's work that Jacob just talked
52:51
about we are also currently starting to build models that are built on a data so Gruta here is leading this effort because kids don't learn from pre-t toiz text right text split up into little pieces that's not how they learn they learn from audio data from continuous speech streams and back in the 90s we as cognitive scientists learned that infants are able to do this because they have very powerful statistical learning mechanisms so they can learn to detect structure in the patterns of this
53:20
continuous speech to detect chunks which can correspond to to words and sometimes to phrases and for those of you who have raised kids you may remember how kidss sometimes will kind of underp parse things they may think that a whole chunk is just kind of one word like look at you they may think it's one thing and this kind of overp parsing and not chopping language into the smallest possible pieces has been argued to be really important to the kind of efficient learning that human infants do
53:46
um and so uh trying to train a Transformer like architectures from raw audio may tell us whether this is indeed going to lead to something like more efficient language learning um in addition to evaluating these models on language tasks like what changu was doing we can also do the kind of things that Jim was talking about to relate these um representations from these developmentally plausible models to human brain data okay so um when we first uh started looking at whether
54:14
language models can capture human responses to languages in collaboration with Josh back in 2001 we published a paper 21 2021 2021 thank you okay I'm recovering from a cold I'm excused um so um we found that a whole bunch of language models like gpt2 bird the kinds of models that you first heard about when they got good they could capture human responses to language but a lot of people said well what what do we make of this these models are so different they're trained on so much
54:43
data doesn't really tell us anything that they arrive at similar representations and so ebal hus who is here somewhere um uh did a very nice study manipulating the amount of data that a model is trained at and so how how does it affect the ability of that model to align to a brain representation right and he found that actually you don't need that much data by about 10 million tokens which is you know really tiny by current standards you already arrive arrive at very good alignment
55:11
with humans um it's like a two two or threeyear old maybe it's yeah exactly a fewy old K gets about that exactly yeah and and now um and so egal was just taking text based models and varying amount of data but now that we have these more developmentally plausible models like model mod that learn both uh from linguistic inputs and vision or models that learn from raw audio inputs um we can try to see how those developmental models are able to capture neural responses to language and not
55:38
just in adults either all that we've done so far was trying to take an adult mature language system and try to use these models as models of language processing in adults but some work that Haley Olen is starting to do a post do between Rebecca and me is trying to track changes in representations in kids across development so you can imagine taking a model and training it in some constrain way and we can of course make the model do you know we can exactly control what kind of input we get how
56:06
how we present the training data and so on and then see which of these models recapitulate the changes that happen as a kid is learning a language because a lot is changing during those couple of years right kids continuously learn a ton of new words a ton of constructions and so that landscape of language knowledge is very very quickly evolving and we're trying to see if we're able to capture those changes during that early development so this is a good segue to
56:31
um talking about what's next so uh uh let's turn it over to Rebecca each of us will talk about like what are some of one one or two examples of like things that are on our our road maps in our part of our missions and the quest that are trying to move the cycle forward yeah it's so hard to choose because there's so many exciting things coming but I thought I'd mention because I told you about um baby brains and the responses to faces that's in a domain where we think babies um start right
57:01
away right looking at other people and connecting to other people is the one survival relevant thing that a baby has to do at the day that it's born and babies do do within hours of birth they look preferentially at faces and they recognize their own mother so that was a case where we were studying something where the architecture had to be there right away what I'm excited about in the coming two years projects that are already well launched is studying the neural basis of um capacities that
57:23
change dramatically in early infancy um so the two of those are navigation and language in navigation we again know a lot from Nancy kemer and others about the neural basis of the perception of scenes the environment around us in adult brains um and in particular a division of labor between the um par cample Place area that tells me I'm in a um seminar room and the OPA which tells me where the way out is um so we've been looking to um study the origins of those
57:49
representations in babies from an age where they can't move their bodies at all they can't sit up or roll over to an age where they begin to walk and have to make navigational choices for themselves and I saw the first tiny bits of those data last night so this is like just about to happen um do that by like seeing movies or something right Show movies of navigation from the first person perspective so and there's huge variability anybody who's had a baby you
58:13
know there's huge variability in when babies start to navigate right my sons could walk before they were N9 months old which was not a good thing because they would walk off of things um and other babies you typically developing babies don't learn to walk until they're 14 months old so there's massive variation in their experience of navigation that we can tap into to study how their visual systems and their intelligence develops similarly in the case of language this is the um language
58:38
system that EV has so beautifully characterized for the last 20 years and through our joint poock kayy Olsen um we're asking what does the language system look like as you're first acquiring a native language scanning toddlers is the craziest stupidest thing anyone has ever tried try to imagine asking a one and a half-year-old to lie still for an hour while they watch your stimuli um but it's incredibly important because there again there's massive variability in the acquisition of
59:03
language healthy kids can range in their vocabulary size from 50 words to 5,000 words at the same age and also some of that variability May predict important differences in School Readiness and in cognitive development and and in language skills for the rest of their lives and this is another thing where the complete data set may have just been collected maybe last week um I don't know the answers yet but so these are I think big results that I hope to have within the next couple of
59:28
years speaking of toddlers um so again so many projects to choose from I chose one that is a little outside uh distribution um because I don't usually work on language but um uh I had the opportunity recently to look at word learning and and revisit a project that we first tied 10 years ago and failed so there've been you know you know gazillions of Corpus studies of early word learning parent checklist of vocabulary so we know a lot about what children say when they say it um uh even
1:00:03
whole wired houses so we see the input to Children's words but what I nobody I think for very sane reasons has tried to ask is given that children know uh you know between 18 and 24 months some 50 to 100 words how do they decide what to say by the time a child's a toddler they have a life right they have bedtime routines and favorite toys and favorite stories and relationship ship with other people and they as we know know a huge amount of the world before language but
1:00:30
they only have 50 to 100 words so it's a huge information compression problem what are they going to talk about and what are they going to say now we know as adults that if we have limited time and limited things to say we are going to tend to talk about the things you do not already know if I go to a coffee shop with you and you step out to take a call and a kangaroo walks into the coffee shop and then walks out the back door and you come back in I'm not going
1:00:54
to say oh the coffee tastes great I'm going to tell you about the kangaroo um so we thought that children might also be selectively informative in this way right from the start so uh we tried that 10 years ago we brought toddlers into the lab and there were things the parent knew and the things the parent didn't know and we wanted to know what the child would say all of these are in the kids vocabulary and what we got was silence because they're toddlers they
1:01:19
were just a handful of them they're in a lab with a stranger when they said things they said mommy or up so there we go which is the way we now have now we have children helping science right um and we were able in parents homes to show toddlers these little movies where the parent and the child see a room with a bear in it and then we say now your parent can close their eyes and cover their ears it doesn't really matter if they're peing as long as the toddler thinks that they
1:01:51
can't see or here right and then the bear puts on shoes right and then the curtain closes and we say can you tell your parent what happens or the parent and the child see that there's a duck right and the parent closes their eyes and a ball comes in again what is the child going to say between 18 to 24 months children know these words they know bear and shoe and Duck and ball what are they going to choose to talk about now if you're an engineer that bar is not going to look very impressive to
1:02:20
you but let me tell you those data points were hard one we had 200 children six trials each so potential 1,200 utterances of which we got 90 right um and in those 90 we surprised ourselves this is ongoing but so far children are systematically talking about what the parent does know they are not mentioning the unknown thing they're not being selectively informative they're talking about what is in common ground and this was really surprising to us but any of
1:02:51
you who maybe has tried to get by in a new language knows how much easier it is to talk about the things that are in common ground the menu or the recipe right or the thing you want to order when you know and if you try to talk about the vacation you took last year or the thing you're planning on doing next week that isn't in common ground it's very hard to communicate communication can break down if you're a new language learner so it may be very rational for
1:03:16
early language Learners to start out talking in ways where they can confirm that bear means what I think it does and you know when I'm saying B right what I'm talking about where if I say you might be like what right you might think you're saying Sho right but it could easily vanish from the communication this is important because we have computational models actually of rational speech act and they are predicated on the assumption that speech is selectively informative that
1:03:43
efficient communication involves efficiently communicating information The Listener does not know and that may be true but rational learning when it's rational communication when it starts out May look very different okay so um you so you so you saw two examples of projects that are in progress developmentally using new tools for studying brains and using new tools for studying infant and toddler and early child behavior on a big scale and again you know because of the tools and the
1:04:11
community that Laura has has built and the quest has helped to built and some of our early quest funders also Jeff halis over there you know really contributed to this um in addition to cbmm and the seagull family and some some of our other great supporters it's you know the hard one work of these couple of individuals is just part of this the Sea of researchers you saw all those studies that are now on our platform so my job as a computational modeler is to now take our model to The
1:04:38
Next Step so working with Rebecca and others and Sher Lou Who I mentioned before we're trying to take that model of infant attention so it goes beyond just the very important first stage that built on Jim's work which is like when do you pay more attention or look longer or Explore More at an interesting object to these Behavior scenes when you see agents even in surprising scenes doing surprising things what is interesting about this when when do you when do you
1:05:03
say why is that person doing that I didn't that's not what I expect where that person could be a little ball right but you know again even young infants will get will get we hypothesize rationally surprised when they see agents doing things that they can't make sense of with their inverse planning model so we're going to see if we can model that using the same kinds of ideas but defined on these probabilistic programs now for our theory of mind or for example again you saw a little bit
1:05:28
of this early building on early things in in what Jim showed before can we go can we build these assist of systems by modeling how people understand what they want from an assistant in terms of both jointly reading our minds and communicating in language so you may not be able to see in the upper right there but a human that's the little humanik robot the the black stick figure character has started in the bottom and taken a couple of steps up and the left and says to the robot who's in the the
1:05:55
hallway on the other side the robot who looks like a robot says can you hand me the red key can you pass me the red key notice there's three red keys and this is exactly how we always communicate we're totally ambiguous and underspecified but if you look at the scene and you know that red keys keys of each color unlock those doors you can say okay well based on where the human is going they must want to go if to a red door and the only one that would make sense is the red door on the left
1:06:20
which is the red key that's closest and most efficient oh it must be number two and that's exactly what people say what our model says notably those large like language Vision models don't do this so this is one place where today's AI systems just fundamentally don't get each other's minds like this but we we think we can build models and then test them on a bigger scale in both adults and also see how these abilities grow as children start to cooperate in the earliest stages of language
1:06:46
learning yeah um cool so to talk about maybe some of the future research projects on the computational side of things coming back to the very beginning of this session one of the things that I really want to emphasize is you know in some ways as little as we understand about or as as many questions as we still have about how human intelligence Works uh we have even more questions about what's going on inside these big models that exist today right uh you know we've seen many examples already
1:07:13
today of sort of surprising failure modes and surprising successes uh made by sort of big deep Network models um and we really as a research Community have very little understanding of what's going on inside those models very little ability to both describe what their internal computations look like to predict how models are going to behave out of distribution um and so one thing that I think is going to be incredibly important going forward and that I'm super excited to work on you know in in
1:07:38
connection with the quest is building tools uh that actually help us improve our understanding of these uh deep Network models which you know in some form are likely to continue to be an important part of whatever computational uh models of intelligence we we build in this effort going forward and so what I'm showing here is uh work by uh couple of folks at MIT including Sarah who was a a quest post talk at some point um on on exactly this on actually sort of artificial intelligence uh tools for
1:08:06
running experiments and sort of producing large scale characterizations of the internal computations inside AI models uh themselves and I think you know starting to have uh systems that can do this that can describe sort of what features are getting represented what internal computations are getting performed is going to be a really useful tool for then figuring out what that next generation of systems ought to look like and I think is something where uh just close connections with
1:08:31
neuroscientists and cognitive scientists is going to be incredibly important uh computer scientists and you know sort of machine learning researchers as a field I think just don't have good languages for describing uh sort of the computations that underly intelligence and for describing the kinds of uh circuits and mechanisms that we might expect to arise inside these models and so uh you know having access to that language having you know sort of people who are used to doing these kinds of
1:08:55
experiments on people I think is going to be really important for understanding uh these automated systems as well um and continuing on that theme of uh sort of using AI tools to automate uh the construction and the understanding of AI models um another thing that I think is going to be incredibly important here is uh is exactly that is using sort of the the systems that we have uh right now uh to help us design and to help us engineer the next generation of systems
1:09:21
um we have seen at various points in in today's presentation I think we're going to see even more in Basha's uh session in the afternoon um sort of explicit symbolic uh probabilistic models of reasoning of computation of planning uh that are able to give us much more robust and much more reliable and much more predictable and Sample efficient um computations for these kinds of cognitive tasks and right now the process of building one of these models for a new problem or for a new domain uh
1:09:50
requires an enormous amount of effort requires you know a sort of very clever grad student or a very clever team of grad students and postdocs and faculty to think very hard for in many cases years and years at a time in order to come up with the models figure out how to do efficient inference in them and run them um and if we want uh the ability to produce these sorts of explicit symbolic models either so that we can put them into self-driving cars and be able to certify that those
1:10:14
self-driving cars are going to do what we want to expect or so that we can fit them to human experimental data or so that we can use them as descriptive models of um neural models we're going to need tools to help us do this and to help us do it better and to help us do it faster and so another sort of major uh effort um uh that has involved a number of Quest members here I'm highlighting Leo and Gabe who are uh I guess current and former grad students here right now um is focusing
1:10:37
specifically on sort of tools for automatically synthesizing models for uh cognitive modeling tasks but I think this is actually a much much bigger area uh and and something with potential applications to many of the other uh kinds of things that can I give my less applied version of that is well I mean then this has been one of this has been one of my biggest uh long-term collaborations with Jacob these are joint students that we both advised and you know one way to think of this is oh
1:11:00
we're trying to understand how natural language can be used as a tool to synthesize code in these probabilistic symbolic World models or the way I think of it as a cognitive scientist is a model of how people understand language and how people use language to communicate thoughts like just what we're doing now so just as you as you heard from the beginning just how do we scale the models inside our own heads so much is not just from our own experience it's from people telling us things so
1:11:21
whether it's a model of how we we build our models from people telling us things or how we can get machines to better Scale Models by us telling them things then it's this is one of the most exciting projects that we're engaged in okay um and that's a good note to turn over to F yeah uh yeah just uh very briefly um one big question that Still Remains is um what is the structure of thought what is the ontology of reasoning right I've put all these different pieces there but it's not like
1:11:48
each of them draws on a separate system on the brain in the brain but it's also not the case that thought is just this nebulous unru structured set of computations so there's been a few systems that were quite um carefully characterized over the years so one system that highlighted in green was a system that Rebecca discovered and characterized A system that supports social reasoning all of the blue things the dark blue things um correspond to A system that supports formal reasoning
1:12:13
formal abstract reasoning and things like mathematics and logic and computer programming all load on that system and all of these are distinct from language again right these are this is just now thinking about the structure of thought and then there is a system for um episodic memory remembering the past and Imagining the future the the Ping but then just in the last few years a few other systems were discovered and this kind of gave me pause because I thought
1:12:36
we kind of figured out the rough Cuts in that space suggesting that there's many other pieces that we may not know yet about so for example um work in ny's lab jointed was Josh has been characterizing the system for reasoning about the physical world um uh also work at ny's lab is finding um a region that seems to be specific for causal reasoning which is of course obviously a very very important piece of reasoning that we rely on starting early in childhood and then hope Keen in my group is
1:13:02
discovering what seems to be a region specific for logical deductive reasoning which is obviously something that is likely well is likely human specific so uh ontologically um uh probably developing quite late but this is all to say that there's a lot of pieces that we just don't even know how how they are implemented whether they are fully distinct or partially overlapping drawing on the same representations and a big open question is how to build models that Implement these different
1:13:31
pieces of um cognitive Machinery that humans have and on our side on the cognitive side understanding how they all develop um and then I guess um uh a big uh part connecting it back to language is understanding how does language interface with all these systems because as I said language is so deeply embedded in everything we do in everyday life trying to understand that interface how do we continually translate between these abstract structure representations supporting
1:13:56
thought and noisy kind of linguistic imprecise vague um descriptions right we do that all the time back and forth quite effortlessly and across a bunch of different domains we trying to understand how might that happen and here we're here we're talking about basically like the the neuros symbolic models are the models that you just saw from Jacob that we've been building and we think these could be the models of how the language Network in the brain and these other networks potentially
1:14:19
this one one can hypothesis how they might interact and interface and grow thought Okay so we just have a couple of minutes and we'll just turn to our last item um feel free to just talk if you want or I have your slide contributions if you want but what what what where do we see the impact in the real world from this kind of work um I don't know I think EV maybe if you want to or um or maybe Jacob first a few of us very interested in education side of yeah I
1:14:44
mean the first piece of this is that and coming back actually to several of the slides that we saw before about sort of theory of mind and inverse planning um these things are especially important once we start to think about about uh you know systems that are not just uh good at making predictions about humans but that are actually interacting uh with people especially in pedagogical settings and so my student Alexis Ross uh lately has been thinking a lot about
1:15:08
sort of AI tutors uh and ways of uh sort of offering personalized education to individual Learners and settings where that's not otherwise available um and it seems to be the case that the real bottleneck there right now with sort of all of the pure llm based systems that we've been trying to build um is that they're actually quite bad at drawing inferences about what it is that individual Learners know or don't know and modeling sort of the progress of their knowledge over the course of a
1:15:31
tutoring interaction and so um a lot of what we're thinking in this direction that I think can build on a lot of the sort of explicit uh computational modeling of theory of mind that other folks are doing here um is using these as actually mechanisms for building tutors that uh personalize their their tutoring strategies to individual Learners and really understand what people know all right EV I didn't have a slide from you but do you want to say about any I just yeah I just want to add that
1:15:57
um these models are also a really exciting tool to understand how things break and how to fix them right because humans break in very messy ways like when a stroke happens it doesn't follow precisely a particular pattern of functional divisions in the brain but using these models once we build a system that we understand well enough what the pieces are and how they go together we can precisely simulate different kinds of impairments by lesioning different components of the
1:16:21
models using the kinds of approaches that have been long used in animal um uh literature which we just can't do with humans but with models we have that ability to impinge on to interfere with models in all sorts of controlled ways to try to understand what exactly leads to problems and once we understand what pieces are critical we can try to understand how to potentially fix them to bring them back online or to leverage existing remaining pieces to lead to the
1:16:45
same kind of of performance like how how our brains get set up education and and how to keep them from breaking on the Ed yeah no but I mean no those are two key impacts and like you know people often talk about in many areas of like education and mental health they say well imagine if it was like Cardiology where we actually had a theory and a model of how the heart works as a pump and understands fluid mechanics and and other things and imagine if we had that
1:17:07
for the mind and brain you know wouldn't I mean well it is worth actually trying to deliver on that and on the education side another direction of yours that I'm really excited about and I think it's something that's a good joint project is like like you know math gets hard when people try to teach kids uh word problems because sometimes you get tangled up is it about the words and the language or is it about the math or the algebra and yeah I mean some of our
1:17:30
Quest teams like Ben Lipkin and others are working on how to understand that better which which we think can lead to much better you know not just just everything better when in in the parts of stem education that uh are most challenging for kids um one more thing about education and then something back about mental health so what I was going to talk about you know it's I think not especially glamorous but it's eminently practical um at the moment and something we're
1:17:56
actually about to try to do um so uh uh I mentioned earlier the enormous difficulty of doing longitudinal testing and pre and post- testing and how very difficult it uh has been actually to look at the effective interventions and um uh one of the peculiarities right now is about 70% of preschoolers are are running around with these pretty high production value very affordable very scalable educational apps and we don't know which if any of them uh work um or
1:18:30
have an impact on children's learning um except for some small scale studies and even those get lost in academic papers so um we have this tool right now children helping science which I think we can turn into science helping children where there's no reason not to conduct rapid Cycle reproducible open access randomized control trials on which of these work because uh uh some years ago at Goan came out with an article in the New Yorker where he pointed out that if we did better care
1:19:00
of uh high blood pressure and diabetes management we would have a bigger impact on public health than any new drug or Discovery ever could make it's incrementalism that would lead to the biggest Innovations and I think it's likewise true here that if any of these work um they are unlike I don't think any of them are going to have the impact of a caring teacher or parent those relationship based deep uh connection based to learning um but those are not uh uh affordable and scalable
1:19:29
interventions these are in their in fam's hands so if we do if any of them were effective those easily adopted interventions could actually make a big Nationwide or International impact on things like kindergarten Readiness so this is a project we're actually just going to uh start and to be clear this is not about AI we can do this with the tools we have right now but if we are going to use AI to build new education uh interventions we need these kinds of benchmarks we need this infrastructure
1:19:59
and Community present to be able to test their efficacy to know what works so we can start here and I hope uh in uh you know next time I I get to talk you I can tell you not only what works today but why we think the future of uh educational interventions may be even more promising and and every one of these apps and every game really a sense is a new lab right a new lab for studying cognition and that's that's you know I think that's like PE people like math apps and they're fun because
1:20:28
they're they're they're games in some form um so this idea that the tools that we're building can turn in a sense like any game becomes a way to study how the mind works any to any tool for teaching is both a way to show that our theories of learning are actually doing work and a way to help keep growing and testing those theories so we really are excited about what that what that potential can look like uh going forward for an understanding of how our minds work and
1:20:54
learn to to actually you know make us have ai that can actually make us smarter so that's a good note to to end on I think and um really appreciate your uh joining us for this session next up is going to be lunch and some really awesome demos you'll get to see some of these things being demoed by the people who built them as well as some of the demos of the embodied intelligence uh uh robots perception things that you're going to see later in the afternoon so
1:21:21
that's probably actually the best part of the day will be the the demos from our students post dos and research staff so make sure you make the rounds and and see them okay thanks to my fellow speakers and uh
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat