Skip to main content
Loading Explore guides…
Lifelogging & Memory Storage · Acedra
Home
Explore
Computer Science
Lifelogging & Memory Storage
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
45 segments
Copy
Hide video
0:00
[Music]
0:12
[Applause] imagine if you could record your life everything you said everything you did available in a perfect memory store at your fingertips so you could go back and find memorable moments and relive them or sift through traces of time and discover patterns in your own life that previously had gone undiscovered well that's exactly the journey that my family began 5 and 1/2 years ago this is my wife and collaborator ruple and on this day at this moment we walked into the house
0:50
with our first child our beautiful baby boy and we walked into a house with a very special home video recording system
1:04
them ready okay this moment and thousands of other moments special for us were captured in our home because in every room in the house if you looked up you'd see a camera and a microphone and if you look down you'd get this bird's eye view of the room here's our living room the baby bedroom room kitchen dining room and the rest of the house and all of these fed into a discar that was designed for continuous capture so here we are flying through a day in our home as we move from sunlit
1:47
morning through incandescent evening and finally lights out for the day over the course of 3 years we recorded 8 to 10 hours a day amassing roughly a quar million hours of multi-track audio and video so you're looking at a piece of what is by far the largest Home Video Collection ever made I think that's clear and what this data represents for our family at a personal level the the the impact has already been immense and we're still learning its value countless moments of
2:24
unsolicited natural moments not posed moments are captured there and we're starting to learn how to discover them and find them but there's also a scientific reason that drove this project which was to use this kind of natural longitudinal data to understand the process of how a child learns language that child being my son and so with many privacy Provisions put in place to protect everyone who's recorded in the data we made elements of the data available to my trusted research team at
2:57
MIT so we could start teasing apart patterns in this massive data set trying to understand the influence of social environments on language acquisition so we're looking here at one of the first things we started to do this is my wife and I cooking breakfast in the kitchen and as we move through space and Through Time a very everyday pattern of life in the kitchen in order to convert this opaque 90,000 hours of video into something we can start to see we use motion analysis
3:32
to pull out as we move through space and Through Time what we call SpaceTime worms and this is become a part of our toolkit for being able to look and see where the activities are in the data and with it trace the patterns of in particular where my son moved throughout the home so we could focus our transcription efforts all the speech environment around my son all the words that he heard from myself my wife or nanny and over time the words he began to produce so with that technology and
4:05
that data and the ability to with mine assistant transcribed speech we've now transcribed well over 7 million words of our home transcripts and with that let me take you now for a first tour into the data so you've all I'm sure seen time-lapse videos where a flower will Blossom as you accelerate time I'd like you to now experience the blossoming of a speech form my son soon after his first birthday would say Gaga to mean water and over the course of the next half year he slowly learned to
4:40
approximate the proper adult form water so we're going to cruise through half a year in about 40 seconds no video here so you can focus on the sound the Acoustics of a new kind of trajectory Gaga to water gaga gaga gaga go
5:12
[Music]
5:42
he sure nailed it didn't
5:50
he so he didn't just learn water over the course of the 24 months the first two years that we really focused on this is a map of every word he learned in chronological order and because we have full transcripts we've identified each of the 503 words that he learned to produce by his second birthday he was an early talker and so we started to analyze why why were certain words born before others this is one of the first results that came out of our study uh a little over a year ago that really
6:21
surprised us the way to interpret this apparently simple graph is on the vertical is an indication of how complex car utterances are based on the length of utterances and the vertical axis is time and all of the data we aligned based on the the following idea every time my son would learn a word we would Trace back and look at all of the language he heard that contained that word and we would plot the relative length of the utterances and what we found was this curious phenomena that
6:55
caregiver speech would systematically dip to a minimum making language as simple as possible and then slowly Ascend back up in complexity and the amazing thing was that the that bounce that dip lined up almost precisely with when each word was born word after word systematically so it appears that all three primary caregivers myself my wife and our Nanny were systematically and I would think subconsciously restructuring our language to meet him at the moment of the birth of a word and bring him
7:28
gently into more complex language and the implications of this there are many but one I just want to point out is that there must be amazing feedback loops it's not of course my son is learning from his linguistic environment but the environment is learning from him that environment people are in these tight feedback loops and creating a kind of scaffolding that has not been noticed until now but that's looking at the speech context what about the visual context we're now looking at think of
8:00
this as a dollhouse cutaway of the of our house we've taken those circular fish Islands cameras and we've done some Optical correction and then we can bring it into three-dimensional life so welcome to my home this is a moment one moment captured across multiple cameras the reason we did this is to create the ultimate memory machine where you can go back and interactively fly around and then breathe video life into this system what I'm going to do is give you an
8:30
accelerated view of 30 minutes again of just life in the living room that's me and my son on the floor and there's video analytics that are tracking our movements my son is leaving red ink I'm leaving green ink we're now on the couch looking out through the window at cars passing by and finally my son playing in a walking toy by himself now we freeze the action 30 minutes we turn time into the vertical axis and we open up for a view of these interaction traces we've
9:01
just left behind and we see these amazing structures these little knots of two colors of of thread we call social hotspots the spiral thread we call a solo hotspot and we think that these affect the way language is learned what we'd like to do is start understanding the interaction between these patterns and the language that my son is exposed to to see if we can predict how the structure of when words are heard word affects when they're learned so in other
9:32
words the relationship between words and what they're about in the world so here's how we're approaching this in this video again my son is being traced out he's leaving red ink behind and there's our Nanny by the door you want to go there all right she offers water and off go the two worms over to the kitchen to get water and what we've done is used the word water to tag that moment that bit of activity and now we take the power of data and take every time my son ever
10:08
heard the word water and the context he saw it in and we use it to penetrate through the video and find every activity Trace that co-occurred with the instance of water and what this data leaves in its wake is a landscape we call these wordscapes this is the wordscape for the word water and you can see most of the action is in the kitchen that's where those big Peaks are over to the left and just for contrast we can do this with any word we can take the word
10:39
by as in goodbye and we're now zoomed in over the entrance to the house and we look and we find as you'd expect a contrast in the landscape where the word by occurs much more in a structured way so we're using these structures to start predicting the order of language acquisition and then that's sort ongoing work now in my lab which we're peering into now at MIT this is at the media lab this has become my favorite way of videographing just about any space three
11:09
of the key people in this project Philip de Camp Ronnie kubat and Brandon Roy are pictured here uh Phillip has been a close collaborator on all the visualizations you're seeing and Michael fleschman was another PhD student in my lab who worked with me on this home video analysis and he made the following observation that just the way that we're analyzing how language connects to events which provide common ground for language that same idea we can take out of your home Deb and we can apply it to
11:41
the world of public media and so our effort took an unexpected turn think of mass media as providing common ground and you have the recipe for taking this idea to a whole new place we've started analyzing television content using the same same principles analyzing event structure of a TV signal episodes of shows commercials all of the components that make up the event structure we're now with satellite dishes pulling in and analyzing a good part of all the TV being watched in the
12:15
United States and you don't have to now go and instrument living rooms with microphones to get people's conversations you just tune in to publicly available social media feeds so we're pulling in about 3 billion comments a month and then the magic happens you have the event structure the common ground that the words are about coming out of the television feeds you've got the conversations that are about that those topics and through semantic analysis and this is actually
12:45
real data you're looking at from our data our processing each yellow line is showing a link being made between a comment in the wild and a piece of event structure coming out of the television signal and the same same idea now can be built up and we get this wordscape except now words are not assembled in my living room instead the context the Common Ground the activities are the content on television that's driving the conversations and so what we're seeing here these skyscrapers
13:16
now are commentary that are linked to content on television same concept but looking at communication Dynamics in a different very different sphere so fundamentally rather than for example measure measuring content based on how many people are watching this gives us the basic data for looking at engagement properties of content and just like we can look at feedback cycles and Dynamics in in a in a family we can now open up the same Concepts and look at uh much
13:47
larger groups of people this is a subset of data from our database just 50,000 out of several million and the social graph that connects them through publicly available sources and if you put them on one plane a second plane is where the content lives so we have the programs and the the sporting events and the commercials and all of the link structures that tie them together make a Content graph and then the important Third Dimension each of the links that you're seeing rended here is an actual
14:22
connection made between something someone said and a piece of content and there are again now tens of millions of these links that give us the connective tissue of social graphs and how they relate to content and we can now start to probe the structure in interesting ways so if we for example trace the path of one piece of content that drive someone to comment on it and then we follow where that comment goes and look at the entire social graph that becomes
14:54
activated and then Trace back to see the relationship between that social graph and content very interesting structure becomes visible we call this a a co- viewing cque a virtual living room if you will and there are fascinating Dynamics at play it's not one way a piece of content an event causes someone to talk they talk to other people that drives TuneIn Behavior back into mass media and you have the cycles that drive the overall Behavior another example very different
15:24
another actual person in our database and we're finding at least hundreds of not thousands of these we've given this person a name this is a pro amateur or proam media critic who has this High fan out rage a lot of people are following this person very influential and they have a propensity to talk about what's on TV so this person is a key Link in connecting mass media and social media together one last example from this data sometimes it's actually the piece
15:54
of content that is special so if we go and look at this piece of content President Obama's State of the Union Address from just a few weeks ago and look at what we find in in the same data at at the same scale the engagement properties of this piece of content are truly remarkable a nation exploding in conversation in real time in response to what's on on the the broadcast and of course through all of these lines are flowing unstructured language we can X-ray and get a realtime pul of a Nation
16:29
real time sense of the social reactions and the different circuits in the social graph being activated by content so to summarize the idea is this as our world becomes increasingly instrumented and we have the capabilities to collect and connect the dots between what people are saying and the context they're saying in what's emerging is an ability to see new social structures and dynamics that have previously not been seen it's like building a microscope or telescope and
17:01
revealing new structures about our own behavior around communication and I think the implications here are profound whether it's for science for Commerce for government but perhaps most of all for us as individuals and so just to return to my son When I Was preparing this talk he was looking over my shoulder and I showed him the clips I was going to show to you today and I asked him for permission granted and and then I went on to reflect isn't it amazing this entire database all these
17:36
recordings I'm going to hand off to you and to your sister who arrived two years later and you guys are going to be able to go back and and re-experience moments that you could never with your biological memory possibly remember the way you can now and he was quiet for a moment and I thought what am I thinking he's he's 5 years old he he's not going to understand this and just as I was having that thought he looked up at me and said so that when I grow up I can show this to my
18:02
kids and I thought wow this is this is powerful stuff so I want to leave you with one last memorable moment from our family this is our the first time our son took more than two steps at once captured on film and I really want you to focus on something as I as take you through it's a cluttered environment it's natural life my mother's in the kitchen cooking and of all places in the hallway I realize he's about to do it about to take more than two steps and so
18:33
you hear me encouraging him realizing what's happening and then the magic happens listen very carefully about three steps in he realizes Something Magic is happening and the most amazing feedback loop of all kicks in and he takes a breath in and he whispers wow and instinctively I Echo I Echo back the same and so let's fly back in time to that memoral
19:05
moment hey come here can you do
19:14
it boy can you do
19:21
it my's walking [Music]
19:29
thank
19:35
[Applause]
19:53
you when I think about succeeding in business I think there's a couple of considerations first and foremost is teamwork today's problems are just too complicated to be solved as an individual and the opportunity to work with others collaboratively where you can build on each other's ideas I think is particularly important I think a second area that's quite important is the understanding of different disciplines it is critical that one be an expert in your major but it is
20:20
absolutely essential that you have the ability to understand where other disciplines the input from other disciplines and be able to incorporate that to make effective decisions and then last but not least uh we are in a global business community and so the opportunity to understand cultures different cultures around the world to be able to incorporate some of the learning from those cultures and to incorporate that into your business decisions is essential to success I have
20:47
three degrees from Cornell so I'd say just about everything that has prepared me came from Cornell and I am indebted to the university for that experience but as I think particularly about my business school career I think the exposure to a variety of disciplines the wealth of resources across the university were exceptionally helpful
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat