Skip to main content
Loading Explore guides…
Future of Computational Linguistics · Acedra
Home
Explore
Computer Science
Future of Computational Linguistics
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
69 segments
Copy
Hide video
0:00
(bright music)
0:03
- This is Stanford Engineering's The Future of Everything, and I'm your host, Russ Altman. If you enjoy The Future of Everything, please follow or subscribe on your favorite listening app. You'll hear about new episodes and it'll help us grow. Today, Professor Christopher Manning, Chris Manning, will tell us how the intersection of linguistics and computer science has led to the remarkable progress in intelligent agents such as ChatGPT. It's the future of Computational Linguistics.
0:34
Before we jump into this episode, a reminder and a plea. Please rate and review the podcast. It'll help us improve, and it'll spread the word, and then you'll know that The Future of Everything will never surprise you. When I think about linguistics, I think about the study of old languages and how they are related and how they teach us about culture of humanity. I don't always think about computer science and its role in these old languages. However, recently, we have seen the rise
1:05
of these intelligent chatbots, like ChatGPT 4, which shows remarkable capabilities in understanding human language and generating responses to a variety of questions across all areas of human endeavor. Are these chatbots a surprise, or were the computational linguistics experts who've been studying this field for decades totally expecting us to achieve this capability?
1:33
Well, Professor Chris Manning is a professor of linguistics and computer science at Stanford University. He creates computational methods for studying linguistics and computational methods for having computers and humans interact. He will tell us that these capabilities were shocking, even to the experts. Yes, there was some early work that anticipated progress, but what we've seen in the last year is something that nobody expected. So Chris, you're a professor
2:02
of both linguistics and computer science. Now, some people might be surprised to know that that's even a thing, because we think of linguistics as the study of languages, old languages, new languages, emerging languages, and we think of computer science as a very different study. But obviously, they're not that different. So could you start out just telling us what is the intersection of linguistics and computer science, and why do we care about it?
2:27
- Well, yeah. So linguistics is a very diverse field. I mean, a lot of the time, people taught think first of philology and reconstructing ancient tongues, but lots of other things go on. So there are socio linguists who look how different communities speak and use language in quite different ways, for example. But for me, what I do is deal with how we can get computers to understand, generate, learn languages, which connects into cognitive science questions 'cause that's also what, variously,
3:03
psychologists would study from a more sort of human-centered cognitive perspective, is how does this go about, where I'm on a slightly more technological level of wanting to get our computers so that they can understand us in the same way that other human beings do. So a lot of that has a more machine learning engineering flavor of how to build models, but it also centrally depends on the subject domain. So just as for your own work, Russ, if you're working in bioinformatics,
3:34
it's useful to know something about biology, as well as something about computation. Similarly, there's value in understanding what the structure of human languages is. - Yes, and I'm sure we're gonna talk about artificial intelligence and some of the amazing things that we're starting to see in a moment. But before that, I'd like to explore a little bit the relationship of language to intelligence. Do you think about it as there's an intelligent being,
3:59
and that intelligent being creates language to communicate? Or is it much more kind of rich connection between how we think about things and the words we have even to think about? So talk a little bit about the relationship between language and intelligence. - Sure. I should preface it by saying this is certainly an area that's been debated a lot by philosophers and cognitive scientists, and not everyone has exactly the same view, but my personal view is that language
4:30
is extremely important to human intelligence. That if you compare humans with some of our nearest neighbors, chimpanzees, bonobos, and things like that,
4:43
you know, it's kind of hard to differentiate us on some of the basics of intelligence. It seems that, you know, you can look at things like planning, tool use, memory, a lot of the things that people talk about for intelligence, and, you know, to first approximation, it doesn't seem like there's much difference. I mean, in some areas, you know, chimpanzees have better short-term memory than human beings do, in fact. But, you know, nevertheless, then it just seems like there's this night
5:13
and day difference between chimpanzees' intelligence and human beings' intelligence and what we've been able to do with that, right? You know, it's the difference between having a stick to dig out some ants versus having a cell phone in your pocket. Seems kind of different. And my belief is that the development of human language has been essential to leveling up human intelligence. That, you know, one side of human language is communication, and we can get back to that,
5:48
but the other side is that human language gave this transformative tool for humans to think with. So it's not that that you can't think without language. You can sort of think with images, as people sometimes do in their dreams. And obviously, you have sort of immediate gut responses when you see something that aren't anything to do with language. You just look at it and you feel excited or repulsed. That's thinking without language. But we humans do a huge amount of structured thought,
6:21
planning, consideration of alternatives in our head in a linguistic sense. I mean, all of us, you know, play out scenarios using words and thoughts in our head. And I think that has been essential in sort of structuring and advancing human thought to allow the kind of higher level intelligence and the results of that that we see everywhere around us. - Great, so you've set up my next question so perfectly. It's almost like you knew what was coming.
6:49
So the elephant in the room, I think it's fair to say, is the recent release and discussions about ChatGPT and other so-called large language models or foundational models. And you've been working in this field for decades. And so one of the things I wanted to ask you is ChatGPT for many people came as a shocking surprise, but I'm wondering, from somebody who's been in the field, is this really a surprise, or have we been making slow and steady progress for this over the last two decades,
7:20
and it's just the entirely predictable result of the research that you and others were doing? - It was a shocking surprise. - [Russ] Oh my goodness. - (laughs) Yeah, so I mean, there are no doubt that you can sort of paint out a history of the progress of research, and that there were different steps along the way. And you can look back and say, "Well, you know, people started using language models. These are models that sort of predict the next words in the sequence around the sort of mid-70s."
7:56
And they started to show that they were useful for speech recognition and spelling correction, machine translation. And it was about 2000 that people started to use neural language models and that they showed some advantages. And then for some of the kind of architecture of these neural networks, you can pick out different components. So I mean, these current large language models are all used in neural models. It's called the transformer. That, you know, it has components inside,
8:22
the residual connections and fully connected layers and attention layers. And you can point to all of the places that they came from from prior work. But despite that all until, let's say, 2017, people realized that language models had an important role for sort of, you know, fluency of text, predicting what's likely in speech recognition, but no one thought this is gonna be the way to achieve language understanding or to achieve the ability to generate whole passages of text, right?
9:00
Tell an entire story. It was sort of seen as for low-level stuff of sort of predicting a few words around each other. And, you know, so it was just very unexpected that this direction that emerged in 2018, that if you just make these neural language models very big, they just start to generate amazing capabilities. And at least so far for the trajectory that we've been in the last five years, as you make these models bigger and bigger, you just start to see more
9:36
and more amazing capabilities emerging seemingly from nowhere. So people often talk about emergent capabilities, meaning that we are just building this bigger and bigger word prediction machine, and yet suddenly these models start having a lot of knowledge about the world, knowledge about human languages, ability to do things like translate, summarize, et cetera. And, you know, I think everybody in the field hadn't expected that, and it was just surprising how this started to happen.
10:10
And so then it's sort of been a gold mine. 'Cause once you've found where the vein is, you keep digging as fast as you can in that direction. - Yes, great. So from your perspective, and so I've seen reports with literally hundreds of pages of examples of remarkable things that this ChatGPT 4 in particular can do. Like, somebody asked it to do a mathematical proof, but express the proof as a poem that rhymes. And it did that. It created a rhyming poem that made a mathematical proof.
10:39
And a lot of these, you know, you could say that they're parlor tricks, but I think that's doing probably injustice to the technology. From your perspective as a computational linguistic, and I know that this is a hard question, what are the one or two capabilities that you're most impressed by in these large language models that you've seen in the last few months? - Yeah, so I mean, yeah, I do think the sort of more kind of fun examples you see in the newspaper really do show the ability
11:08
of these models sort of to put stuff together in creative and clearly original ways. So, you know, a lot of them are fun, and I don't think of them as parlor tricks, but, you know, in terms of what seems to be special,
11:27
I mean, quite a bit of what I'm seeing as special in the most recent models is that these models do seem to be starting
11:39
to develop the beginnings of a model of the world, where they're maintaining the scenario in their head and can reason with it. So, you know, early on, people used to say, "Oh well yeah these models are very good at completing sentences," and that they can tell a bit of a story. And well, by the time they could write a five-paragraph story and it actually made sense, it can kind of continue along coherently in a plausible way that was interesting to read with creative details,
12:13
that to me was already amazing because, you know, like, not that long ago, we thought we were doing well in natural language generation if we generated one reasonable-sounding sentence. The idea that we could continue through sort of 20 sentences in a five-paragraph story, and then all follow from each other and make sense, that was sort of a complete, that seemed a completely out of sight ability for natural language generation. But we are now getting more than that, right?
12:43
That in the earlier smaller models, that if you set up scenarios where the model had to kind of maintain a good understanding of the world and be able to reason about it so that you've started with some facts about, you know, John knows some facts about their mortgage, and John's afraid to tell his wife because she will be concerned about X, and then John tells this other person. And so you're starting to put together this complex world model that the language model
13:26
is also putting together some kind of model of the world. So you can then ask inferences based on that world model as to when the friend meets John's wife, what should they do, and what concerns will they have. And the model can answer with the kind of same kind of ability to reason about situations as a human being could. - Of course, we can't go into the details of how these models are built, and I don't want to, but they have seen incredible volumes of human-generated text.
14:03
And as you're saying, they seem to have learned a lot more than we would've expected about even human relationships and how they work, and how kind of unstated motives might play out in a scenario. Let's go to the topic, and this is, I'm sure, where you're getting your research agenda for the future. What are the things that it's not doing well, or where are the things that we really need to focus attention as these things continue to be rolled out and really made available? I think I've read
14:33
that it's been the most quickly adopted technology in terms of the number of people who have signed up to use it, either in their personal life or their professional life. So what should we be worried about? What is it not doing well that might not be in all the advertising material? - (laughs) Fair enough. Yeah, so perhaps the first thing that's been very represented in people saying, you know, be worried about these models is, I mean, it is commonly called hallucination.
15:03
I think that's sort of a bad term, but these models will just make stuff up. Now, you know, there are some humans that just make stuff up. We probably- - They're often in the news. - See a few of them, yes. 2017 to '20 or something. But, you know, in general, human beings have a pretty good sense of what they know, and in certain circumstances, you know, they'll tell a yarn. But, you know, most of the time, they know what they know and will be reporting truthful things in that space.
15:40
It's actually kind of a deep architectural fact about how these models are designed, is that the whole goal of them is given this context, predict what's most likely to follow. And that means if they know facts, that they will basically give those facts. But if they don't know facts, or you give them some kind of counterfactual scenario, they will with equal confidence in the way they talk just put whatever. So, you know, maybe the model doesn't know much about your educational background, Russ,
16:14
so it'll say, "Okay, Stanford professor. Well, maybe that means you've got your PhD from MIT, say. And well, maybe you had your, you know, first job at Columbia, and then you move to Stanford." It'll just write this biography- - [Russ] Just for the record, none of that is true. Right. - As if it were all true, completely straight faced, but it's just making stuff up. So that's one huge problem to solve. And, you know, I think we don't currently have a very good idea
16:45
as to how to solve that, right? You know, engineers are sort of clever at refining things and improving metrics. So, you know, I think we'll certainly see the amount of stuff made up starting to decrease as people do iterate on these models. But it's very central to the current architecture that these models, just in any circumstance, put next what's most plausible. So I think we do actually need some more profound architectural advances. I mean, one of the problems with these models
17:16
is that they are essentially just feed-forward models. So at any point, they're sort of running stuff through a neural network, generating the next word and repeating over. And so something people are starting to experiment on is sort of reflective models, which can kind of sort of, where stuff feeds back into itself again. And, you know, that might give it much more ability to deal with some of this invention. I mean, the kind of interesting fact is, you know,
17:42
if you take one of these great models like ChatGPT or GPT-4 and you show it something it just generated, and you then just ask it, "Do you think this, how sure are you that this is true?" it's actually pretty good at answering that, right? - Wow. - It can often differentiate
18:04
what it just invented versus stuff that is true. And that's because really, internal to its probabilities from its training data, it does have more idea as to, you know, what things are well supported versus what things are made up. So we kind of need to sort of have an architecture where it can be more using that knowledge that it even has at the time it's generating to differentiate things. And so that leads on to this kind of concept I sort of mentioned before of world models.
18:34
So, you know, human beings, internal to their head, have a model of the world. You know, that's exactly what we play out when I say, "Okay, I have to go and talk to my department chair saying I want to go on leave again. Gee, they're probably gonna react. I was on leave two years ago. And so I'd better come up with a good explanation of why it'd be okay for me to disappear for another six months." You know, we have a model of the world and of people in the world,
19:01
and we can sort of use that to plan out and put things together in all sorts of ways.
19:08
Now, these large language models, you know, on the one hand, are starting to develop models of the world. We mentioned that before. And in fact, you know, basically they're now the best computational models of the world that we have. So, you know, people in robotics are starting to use large language models as world models because they actually help them predict actions that will happen in the world with different objects and different people.
19:36
But our current models still have very poor world models. And so somehow working out how to have better world models and maintenance of world models in these models. Yeah, so there's still lots of research directions. We shouldn't give up quite yet. - Great. Well, this is The Future of Everything with Russ Altman. We'll have more with Chris Manning next. Welcome back to The Future of Everything. I'm Russ Altman, and I'm speaking with Professor Chris Manning of Stanford University.
20:03
In the last segment, Chris told us about the complex interaction between intelligence, language, and computer science. He told us that ChatGPT 4 was kind of a huge surprise, even for the experts. In this segment, he will tell us about some of the risks of this technology, and we'll also go back to translation and talk about what is our current capability for translating between languages. And how are we doing for the old languages that are, in many cases, disappearing
20:32
because their speakers are learning other languages instead? I wanna start out in this segment just asking you about what you conceive of as the major risks of these technologies. You talked about what the technical challenge is to making them better are. Separate from that, what could go wrong in our use of these technologies? - Yeah, so there are lots of risks and lots of things that could go wrong. Some of them are sort of very immediate and direct. So these models provide a very cheap way
21:11
to produce large amounts of text, and they can be fine tuned to produce text
21:19
that works the best to influence people, right? So the industry of advertising has been involved for close to a century now, I guess, with writing text that influences people to buy certain products or to vote for a certain person. But, you know, that's been, you know, relatively speaking, expensive work for you to pay to get people to do. And we are facing the chance now that we'll be able to have these models do that kind of work, not only about two orders of magnitude cheaper,
21:51
100 times cheaper, but actually much better, because using our machine learning technology, we can continue to tune these models,
22:01
you know, and have individualized models. So there's a model that's especially good at persuading Russ Altman who to vote for in the next election. And, you know, already there are sort of problems with people being too influenced by, you know, both advertising and populist opinion. And if that sort of is made much worse, that's potentially quite bad for society. - [Russ] Wow. - There are other risks as well. I mean, you know, what some of the other, I'd say,
22:33
you know, that are sort of influencing people. There are other forms of that, right? There's sort of disinformation coming whether from state actors or large companies wanting to influence public opinion. But there's then also concerns that come from, you know, biases that might be built into these models. So the fact is these models are dominated by the people who tend to have power because they're by and large the people who write the most and gets disseminated the most.
23:02
So they're not equally representing all the voices of humanity. And so that is then a bad source of bias, which can further, you know, drive things away from the direction of trying to give us equality in society. And that's especially worrying because at least on the trajectory we are on at the moment, there is a small number of large language models that are dominating the scene, such as the ones from OpenAI. And so if every, well, to first approximation, everybody is using the same models
23:40
and they've got particular biases in favor of certain kinds of people and against other kinds of people, well, it's sort of bad if you aren't on the right side of that equation. - And it sounds to me this is not a purely technical challenge for folks like you and your research colleagues. It sounds like society is gonna have to make decisions about what is and isn't allowed. And if there are any kinds of cones that they want to set up for the behavior of these models.
24:06
And I'm sure that's a difficult discussion. Is it happening?
24:10
- It's starting to happen, but I think it's just not happening with the speed and the level of sort of focus on what are quite difficult technical issues that's really needed. Yeah, I think there's just no doubt that we need to be doing more to think about the consequences and regulate what kinds of things are and aren't okay. But, you know, certainly in the United States, we're in this situation where most of Congress barely understands how current generation social media works,
24:46
let alone having the kind of background for sensibly thinking about how to regulate and control these models. - Great. Well, not great, but thank you for those comments. Let me move to an entirely different area and actually going back to the roots of our discussion in linguistics and computation. I know that one of the big challenges for computational linguistics for many years was translation, translating from English to Spanish, from French to Russian. So let me ask, and it looks like
25:15
that these models are quite good at translation. As a linguist, what are you seeing there? Is this a solved problem? And at the edges, are there still issues that we have to pay attention to? - It's definitely not a solved problem, but enormous progress is being made. Yeah, so I mean, these models are just sort of trained on a lot of text of various languages. So the fact of the matter is, you know, just out of the box you can ask ChatGPT to translate between languages,
25:43
and it does a pretty passable job. It's still the case that people build dedicated neural machine translation models explicitly trained on text to translate, and, you know, they're even better. And so we've reached the point for major languages that, you know, it's not that everything is always perfect, but, you know, translation is just good now. I mean, from those of us at least been around for a fair while and remember the kind of garbage you used to get out of machine translation,
26:15
I mean, now, you can sort of take a paragraph in German, Italian, French, Spanish, stick it into Google translate, and read the English translation and, you know, to first approximation, just be perfect. You know, some other languages like Chinese are still a bit harder. So there's a very good news story. The question is how that extends out to the whole of humanity. So people normally count about 7,000 languages in the world. I mean, you know, a lot of those are unfortunately languages
26:50
that aren't gonna be with us much longer. So for example, lots of languages in different areas of the planet, including Native American languages and Australian languages. You know, they're down to a handful of people. And although there are, you know, some people are working hard to preserve and reclaim those languages, it seems like there's just sort of no doubt that within a century's time, the number of languages will be down to, well, at least 2,000, maybe only 1,000.
27:21
- Wow, that is a huge contraction. - So there's gonna be a huge contraction in the number of languages spoken in the world. But, you know, even if we stick to 1 or 2,000, the fact of the matter is there are only fantastic amounts of data that allow building really good foundation models like we've been talking about, for you know, maybe the top 20. I mean, even within the top 20, there's an enormous difference between the amount of data you can collect in English or Chinese versus the amount
27:53
of data that you can collect in Bangla or Portuguese. And so then things fall off very rapidly get from there. And so there's a real haves and have-nots of this new technology. You know, so it's a mixed story. There's a good news story and a bad news story. You know, the good news story is these models are much better at transferring capabilities across languages than anything we had before. So we're actually sort of making progress. And we're also better able to handle smaller languages
28:30
than the we used to be. But, you know, that's a relative claim. It's still the case that there are lots of languages with millions of speakers for which we just don't have good language technology. And there's no easy way to make it even in the current world because we just don't have the kind of data resources to be able to do so. And that reflects things like the legacy of colonialism, right? So you see in much of Africa that, you know, there are major languages in Africa
29:04
which will have, you know, 10 million plus speakers, lots of people, right? So those are larger languages than, you know, European languages like Danish or Norwegian. But the fact of the matter is that the educational system in those countries, that people are still, you know, being schooled in English or French, right? So these are sort of languages of the community. And because of that, there just isn't the available resources in terms of written materials, et cetera,
29:34
to be able to build the same kind of language technology. - So in the last 30, 40 seconds, I did want to ask, is there any way that these language models might help this problem? I understand all the ways in which kind of social forces have led to a loss of languages, but is it possible that they will have such a deep understanding of how human language works that with relatively small bits of language from these dying linguistic traditions,
30:03
that we might be able to like regenerate the information? Because we understand so much about language that just given one book or one kind of, a piece of text, that they'll be able to be... I'm thinking of like "Jurassic Park." Like, in "Jurassic Park," they get a little bit of DNA from the T-Rex, and they're able to regenerate the T-Rex. Is there gonna be anything like that that you see on the horizon for like forensic linguistics?
30:31
- Yeah, I think so. And it is already the case that neural models have been applied to deciphering and decoding languages. I mean, I think you can only go so far, right? So you're more expert on DNA than me, but, you know, like, if you have the DNA, that actually is giving you the whole blueprint, so you only need, you know, a few strands of DNA. And if you've got good enough science to decode it all, you've got everything, in some sense. Whereas if you've sort of only got one book
31:02
or some short stories, you just don't have nearly enough of the language. You just don't know what other words are. So anything that you sort of do in those circumstances can try and be faithful to what you do know and exploit that, but also has to invent stuff. And, you know, that's what's happened in human cases as well, right? So that when Hebrew was in revived for modern Hebrew, well, you know, certainly, it was following what was in ancient Hebrew, but, well, people needed a lot of words
31:37
for stuff that they didn't have words for. And so a lot of new stuff had to be invented. And, you know, more extremely, the same as being done in some cases of reviving things like Australian Aboriginal languages, which is only sort of quite partial historical documentation. So you are trying to follow the genius of the language, but you have to be sort of filling, coloring in stuff that is sort of plausible, but can't actually claim as fact. - Thanks to Chris Manning.
32:06
That was the future of computational linguistics. You have been listening to "The Future of Everything with Russ Altman. If you enjoy the podcast, please consider subscribing or following it on your favorite app. You'll never be surprised by the future of anything. Maybe tell your friends about it too. Also, of course, rate and review it. It will help us grow, and it will help us improve. We have more than 200 episodes in our archives from interviews with people who are inventing the future.
32:34
Consider checking those out as well. You can connect with me on Twitter at Rbaltman, and you can follow Stanford Engineering at StanfordEng. (bright music)
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat