Skip to main content
Loading Explore guides…
Selective Attention in Psychology - · Acedra
Home
Explore
Psychology
Selective Attention in Psychology -
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
134 segments
Copy
Hide video
0:00
ROBERT DESIMONE: So the type of attention that I'm going to talk about today is what we call selective attention or the type of attention that allows the student to concentrate on the task at hand-- for example, the book in front of her-- and block out the distractions around her. And I'll go through some of the computational reasons for having an attentional system like this. But I want to give you some demonstrations of what a profound effect attention can have on our visual awareness.
0:37
You may have already seen demonstrations like this. But just excuse me if you have. But if you haven't, this is meant to impress you that studying attention could be worth studying. So what you're going to see in the next slide is a complex scene blinking on and off. And all I want you to do is mentally note when you notice something changing in the scene. I hope everyone can see the left scene flashing on and off. And some of you-- again, I can't see you, but hopefully some of you
1:15
are maybe beginning to notice that there's something changing in the scene. And if you haven't by now, I will give you a hint that it's the airplane engine that's blinking on and off.
1:36
This is called a change blindness demonstration. You didn't pay attention to it. The transience of the screen going on and off kept your attention from being attracted to the transience of the airplane engine disappearing. But I am positive your eyes passed over the plane engine even when you weren't noticing it. So I'll give you another demonstration. This is called a mudflash demonstration in case you don't like the flashing screen. Here, the mudflashes keep your attention
2:07
from being attracted to transience. Does anyone see anything changing in this scene?
2:17
Well, here's another hint. It's the line down the middle of the road, which is changing from solid to striped, on and off. Again, something that's stimulating your retina, but since your attention wasn't attracted to it, you basically weren't aware of it at all. And then lastly, in case you think flashing is some sort of trick, I'm going to give you a demonstration from Aude Oliva at MIT, which just has a complex scene that changes gradually. So the same thing, just sort of mentally
2:50
note when you notice the scene changing.
2:59
When you see something change in the scene.
3:11
OK, so these are all the things that changed. I'm not sure if you caught all of these. There were signs, people, doors, windows. All kinds of things changed in the scene. But because they were changing gradually, they didn't attract your attention. And again, you basically didn't consciously process this. So these are all changes occurring on your retina. But some place between your retina and your soul, they got filtered out. So these attentional systems, besides having
3:46
important goals in visual processing, also affect our awareness. So how do they come about? Well, one thing that many people imagine is that attention is just the same thing as looking. In other words, where you look is what you attend to. And if you look at people's scanning eye movements, as is shown in these examples here, where you can see the picture and then where do people's eyes go, people's eyes go to the relevant aspects of the different pictures.
4:20
And people think that that's what you attend to. But obviously, in the examples I just showed you, we know that your eyes were passing over these important things. And so it wasn't just the eyes. And there's something else that attention and eye movements are normally linked. But they're not the same thing. There's something else. And we'll talk about what this is. And if there's time at the very end, we can talk about the role of eye movements in attention.
4:51
We have some recent work on that that I think you'd find interesting. But let's see how far we get. So-- [INTERPOSING VOICES] --thinking about these kinds of display is that-- [INTERPOSING VOICES] --your attention first on the scene and then the eyes follow. The attention can be quick [INTERPOSING VOICES] natural movement. AUDIENCE: [INAUDIBLE] ROBERT DESIMONE: So why do we have-- why do we need an attentional system at all? And the classical demonstration of why we need an attentional system--
5:23
[INTERPOSING VOICES] --is an example of what psychologist Cherry called the cocktail party phenomenon. Now-- [INTERPOSING VOICES] I can hear-- can you mute your microphone there? Because I can hear all you moving around and so on. If you want to just mute yourself there. AUDIENCE: Sorry.
5:49
ROBERT DESIMONE: So the cocktail party, you find yourself where all kinds of people are talking around you. And if you tried to follow all the conversation of all the people around you, you would find that impossible. You can't process multiple conversations simultaneously. So what that tells us is that, in this case, our auditory language system has some kind of limited processing capacity. It can process some portion but not the whole thing. And then the other flip side of this
6:29
is it demonstrates the other important phenomenon about attention, which is that when you do focus on one conversation-- so you think that some person is saying something really interesting, so you focus your attention on that person. And once you do that, now you can fully process that conversation. But you block out all the other ones. So that's the flip side, which is what we call selectivity. So on the one hand, you have processing limitations. And the other side, you have selectivity,
7:03
which can, in some sense, overcome the processing limitations for at least one datastream. Now, psychologists would love to study attention just by going to cocktail parties. However, they usually take these phenomena into the laboratory and try to study them. And in the laboratory, there's lots of elaborate tests. But many of them follow the principles demonstrated here, which is what happens when you just flash a bunch of letters on the screen and ask people to do something like report
7:42
the red letters. OK. Now, if you compare how long it takes or how many letters people can report, if you compare a to b, what you find is that people will be more successful in reporting the red letters in b than in a. If it flashes up very, very briefly, they may not be able to report all the letters in a. But they very well might be in b. So that tells you that there's this processing capacity limitation. Now, if you compare the time or the performance on a versus c,
8:28
then what you find is that the performance on a versus c are actually the same. So when you do-- and the difference here is that now it's the number of background elements that varies between the two displays, not the number of attended elements. And when you have this difference in background elements, basically, the number of background elements doesn't matter. So you can do equally well in a as in c because you're filtering for just the red letters, right?
9:05
So you've been told to report red letters. You can select out the red letters. The number of green letters just doesn't matter. So that gives you the processing limitation. I see that Arturo [? Daza, ?] your image is on the screen. Does that mean you want to ask the question, or that's just random? AUDIENCE: [INAUDIBLE] AUDIENCE: I just logged in as a co-host, Bob. No worries. ROBERT DESIMONE: What's that? AUDIENCE: I just logged in as a co-host. No worries, no worries.
9:31
ROBERT DESIMONE: Oh, OK, OK. Good. All right. There's another thing you get that could be illustrated by these letter displays, which is the phenomena of pop-out, which is that if you look at the performance on the left versus the right here, the performance of being able to report the red letter on the left is far superior than being able to report the red letter on the right because in the left, the N will pop out. And on the right, it doesn't because now
10:05
there's more variability in the background elements. And the display on the left is supposed to be an illustration of what people call pop-out, that even without instruction, things that stand out from their background will be processed preferentially. So even without instruction, if you just ask a subject to report what letters you see, on the left, they're going to report N. And on the right, they can report anything. So there's some preference for things that stand out from the background.
10:39
We're not going to talk through about this. OK. So we've been trying to understand how this works by studying the object recognition stream, which I assume you've heard about by now in this course, which is a pathway that extends from primary visual cortex through V1 to areas like V2, V4, and then extends into the inferior temporal cortex. And what we know from studies in which people have looked at the effects of attention along this pathway is that attention improves neuronal processing.
11:20
So on the left, at the top, we have a typical kind of attentional display where the dot is the monkey's fixation. The monkey is trained to fixate that spot. And then there's, say, two stimuli placed on the screen. This dotted box is the receptive field of the cell. There's one stimulus placed inside the receptive field, one stimulus placed outside. And then through task conditions, you can instruct the monkey to pay attention to the stimulus inside the field or the stimulus
11:51
outside the field and then plot the response to the stimulus inside the field in both cases. In one case, it's being attended. In the other case, it's being ignored, and something else is being attended. And if you plot just a normal firing rate histogram, what people often find is that the firing rate for the stimulus when it's attended is higher than the firing rate for the same exact physical stimulus when it's ignored. Now, in addition to changes in firing rate,
12:23
we also know that these attentional instructions also change measures of what we call coherence or the coordinated activity of neurons. So one way in my lab that we study coherence is we use multiple electrodes. We record spikes on the different electrodes but also the local field potential. And local field potential is essentially a measure of population activity. And we can look at the timing or phase relationship between the spikes and oscillatory components in the local field potential.
12:58
And if the activity is-- basically, the local field potential is just serving as a kind of time base here for activity in the population. If spikes tend to occur at the same time or same phase on every cycle, then it'll be high coherence. But if they occur randomly with respect to each other, there's low coherence. And when you compute this in a population, you typically get something like what's shown here, which just shows the coherence as a function of frequency.
13:30
And again, this slide and the other slides, red will be the attended condition, and blue is the same stimulus ignored. And you can see there's increased coherence in the gamma frequency range from about 4 to 70 hertz. And that's thought to be within the time window in which cells are normally integrating inputs. So a postsynaptic cell getting input from these cells would see a more synchronized time-like barrage of spikes in the attended condition
14:05
in this time window that's relevant for integration. You also see the reverse effect at low frequencies, particularly in the alpha range, but broader than that. And this desynchronization at low frequencies is also typically found in these attentional studies. But I'm not going to have time to go into that aspect today. OK. So with attention, enhanced firing rates, enhanced synchrony, all of which is going to have the-- [RINGTONE] All of which is going to have the consequence
14:43
of increasing the impact on the postsynaptic cells.
14:49
Now, about this synchronous activity, you've probably heard about it before. And you may hear about it more in the class. What is this all about? And there's been lots of ideas about why the nervous system uses synchronous activity that can be seen in different frequency ranges. In my lab with Georgia Gregoriou, we emphasize that synchrony can help communication across areas. But if you have area A, the cells are all synchronized in their activity,
15:24
and if they synchronize with the cells in area B, then the input cells are going to have a bigger impact on B. It could be the spikes could be arriving at a time when the cells in area B are depolarized. They're prepared for input. That could enhance communication. Pascal Fries has talked about [COMPUTER SOUND] through coherence. And he has raised the possibility that the nervous system could regulate communication between areas by changing the coherence between cells in different areas.
15:59
Earl Miller has raised the possibility of phase coding, where the phase that the spikes occur in itself carries information that the nervous system uses. And in all of these ideas in general, the causal role of synchrony has not really been tested because it's incredibly difficult. But John Reynolds has started to do this using optogenetics. And I think we'll see a lot more studies like this in the future.
16:30
But for now, we will stick with the idea of [INAUDIBLE] synchrony enhancing the postsynaptic input. Now, one of the questions raised when you see changes in synchrony is whether there is something that regulates the synchrony like a master clock or a master controller of synchrony. And it's been known from computational studies that such a master controller is actually not required, that you can get synchrony just given the known biophysical properties of neurons
17:06
and anatomical connections between them. And an example is shown in this next slide, which shows a series of metronomes that you're going to see when you see this video, the metronomes are just ticking away at random phase when they're sitting on this board. When the board is raised and they're set on a couple of soft drink cans, now you have the possibility of one metronome pushing the board and influencing another metronome. And then you'll see what happens in this circumstance.
17:44
[VIDEO PLAYBACK] We start off at random phase.
17:54
- No, I'll move it back.
18:03
ROBERT DESIMONE: You see them gradually going--
18:19
OK. So I've done this demonstration myself. I have these metronomes in my office. I can tell you it's a lot more fun studying physical networks than it is computational networks. You can play around with this. Physicists have examined the properties of synchronized oscillators like this. Actually, this ability of metronomes or oscillators to synchronize was first noted by Huygens in 16-something. He was building pendulum clocks for his ships. And he had two pendulum clocks hanging from a beam.
18:56
And he noticed that the pendulums synchronized with each other. And first, he thought maybe he discovered some new force of nature [COMPUTER SOUND] how clocks could synchronize. But later, he realized that the beam was actually not completely rigid and that is was actually jiggling back and forth with the clocks. And when he nailed down the beam so it wouldn't jiggle, then the synchrony went away. And yeah, since then, people have studied the physical properties of oscillators.
19:26
But all you really need in these kinds of demonstrations we believe in the brain, as you just need anatomical connections that, given other fundamental properties of neurons, can lead to synchrony, which will have an impact on processing. What I showed you previously was what happens when you have an attentional paradigm. You have a stimulus in the receptive field, another stimulus someplace else. You attend inside the field or outside the field, and you get this enhancement.
19:57
But there's a little bit more complicated situation when you have multiple stimuli in the same field. And that's illustrated here. So the paradigm is shown up top. We have a situation where in every case, the monkey just maintains fixation on a spot. So there's no intentional manipulation here. And on the left, we show a stimulus that we call the good stimulus for a cell which elicits a strong response from the cell. Another stimulus is this diagonal yellow bar
20:28
we call a poor stimulus because it elicits a poor response. And then we have a situation where they're paired. So if you go to the histograms below, the red line shows the firing rate for the red stimulus alone. That's the good response. The yellow line shows the response for the poor stimulus alone. And that's this weaker response down here. And then when you pair the two stimuli together, you almost never see that the response is the sum of the response to the individual ones alone.
20:58
It's almost always something like the average. And that's what you see here. The response to the pair looks something like the average. However, when you direct the animal's attention to the stimuli when you have them paired, you get something different. So now we see here in the condition where you direct the animal's attention to the poor stimulus, you still have the pair of stimuli in the receptive field. But now the response to the pair drops down
21:32
to about the level of response you have to the poor stimulus itself. Whereas in another condition, if you attend to the good stimulus, now the response to the pair pops up and is almost as good as the response to the good stimulus itself. And so in other words, without attention, the cells are doing something like averaging. Well, there's variability in this. And you can imagine that without attention, you get a kind of statistical sense of what's there out in the visual field.
22:10
But with attention, we're biasing this competition between stimuli. Then we get more of a pure signal about what that stimulus is, either the red stimulus or the yellow stimulus.
22:26
And this filtering out of distractors takes place at every level of the visual system. Certainly, every time you have multiple stimuli in a receptive field, at least starting in V2 and continues down the ventral stream, the step by step competition and filtering of distractors. So by the time you get to the temporal cortex, you would have largely a representation of the attended stimulus.
22:58
And this general phenomena led John Duncan and me to propose-- now it's a long time ago-- this general idea of biased competition. The stimuli in the visual field are in competition with each other for control over our visual processing resources. Every object's in competition with every other object. And then we simply bias the competition towards objects that have behavioral relevance. Not just us. Animals do this. And the high objects with high bias or high salience
23:31
will more likely win the competition. And then once something is winning in the competition, the distracting information is filtered out. And then the prediction is that object recognition and clutter will be much improved. So to test this idea, we collaborated with Tommy Poggio in his lab with Ethan Meyers and did an experiment to test this. And the general [COMPUTER SOUND] in the experiment was as follows. So as you've probably heard from your lectures
24:05
in the visual system, we believe that objects in the visual field are coded or represented by the pattern of activity across a population of neurons. You can regard this as a sort of normal feature vector with different neurons coding different features of a complex stimulus, like this car. And to represent a different object, you have the same neurons representing different features. But now they'll be a different pattern across the population.
24:38
There's another reflecting the features of a different object. But the problem comes when you have more than one stimulus in the same receptive field. We are now the same neural vectors that represent two different stimuli. And the idea is that you can get some degradation of the code. But with attention, the hypothesis is you should restore the code. That's the hypothesis. And so we did a simple experiment with recording from neurons in IT cortex.
25:17
And we had the animal looking at different objects on the screen. And then we sent the firing trains of the neurons through a pattern classifier, which learned to associate relationship between the neural firing patterns and the object. So you learn different objects, different firing patterns. And then after it's been trained, you can make predictions about what the monkey's seeing, whether it's a correct prediction or an incorrect prediction. You can quantify that. And so we did.
25:50
And so we just asked, what's the decoding performance of this population with and without attention and with or without multiple stimuli? The actual experimental design is shown here. The monkey is fixating a spot. And then we can place either one or three stimuli within the same IT neuron receptive field. In the trial, the animal starts off fixating. There's an array of stimuli that appear. And then a short while later, a cue appears pointing towards one of those stimuli, a very dim bar.
26:23
And that's the cue for the animal to pay attention to that stimulus because, at some random time in the future, that stimulus will change color slightly. And the animal knows that if it then makes a saccade to that attended stimulus, it will get rewarded. And if the other stimuli that are not cued change color, should not respond to those. So that's the basic task. And then we can-- yeah. And then the stimuli came from these classes of cars, faces,
26:57
fruits and vegetables, and furniture, which, as you know, this represents all the major categories of objects in the universe. So we generalize from these.
27:12
And then the way we quantified the results is we just used the time window in which we did the decoding performance and then just slid the time window across the time from the trial. And we used the area under an ROC curve as the measure of performance where 1 would be perfect classification [INAUDIBLE] OK, so what happens when we run this experiment? OK, so when you only have a single object in the receptive field and look at the decoding [COMPUTER SOUND] it starts off at chance.
27:48
But then at time 0, the array appears. And then the decoding performance ramps up. In this particular case, it was over 80%, which is not too bad. It begins to fall off over time. The cue appears here at 500 milliseconds after the array onset. But there's only one stimulus on the screen the animal's visually paying attention to if the cue carries no information. But now let's see what happens when you have the three stimuli. And now the decoding performance is shown now
28:21
by the green and red bars. The red bar is when the animal is decoding. This is the decoding performance for the attended object. And you can see that when the cue appears at 500 milliseconds, the decoding performance for the attended object starts to rise. And the decoding performance for the unattended object actually declines. So the code is better for the attended object across the population, which is just what we were predicting. Now, it's not equal to that of an isolated object by itself.
28:59
But you know that attention is not a perfect filter. That's why I often tell people, if they want to-- people ask me for advice on, how can I more effectively block out distracting information? My advice is always physically turn it off. If you use your brain, it's only going to be somewhat effective. So you're not going to get perfect performance with [INAUDIBLE] These cell responses, they're going. Population responses are changing. Responses are going up and down and so on.
29:32
Where is this all coming from? What causes these changes. And one source of feedback to the object recognition system will start to come from actually what's known as the dorsal stream. Now we have this, I'll show a wiring diagram that shows this object recognition stream down below, the area shown in red. And there's this dorsal stream of areas that are more concerned with space in the parietal cortex and space and objects in the prefrontal cortex.
30:03
And certainly, for spatially directed attention, it's thought that some of these areas in the parietal cortex and the prefrontal cortex play an important role. The feedback from these areas is modulating activity in the object recognition stream. And so we've tried to understand how that works, although we have focused almost entirely on the prefrontal cortex. And so I'll just show you those results.
30:34
So here we see what happens if we record from neurons in the frontal eye fields in the prefrontal cortex, which is thought to be a central area in the control of spatially directed attention. And then area V4, an area-- actually, that was an area that I used an example from previously when I told you about the enhancement of firing rates. That's intermediate area one, that processing stream. And we see the firing rates to a stimulus, in one case, where it's attended-- that's the red bar--
31:06
and the other case where the same stimulus is ignored. That's the blue bar for firing rate. Time 0 is when the cue comes on and tells the animal what stimulus to attend to. So you could see in the frontal eye fields, it fired shortly after the cue appears. The firing rate goes up for that stimulus. Whereas if the animal's cued to something else, the firing rate for that stimulus goes down. And you can measure the latency of these effects. When did firing rate first begin to change
31:39
after the attentional cue? And in the frontal eye fields, it's around 80 milliseconds. This is a population result here. And in V4 population that works about 120 milliseconds. So the first effects of attention are observed in the frontal eye field. And then they occur later in V4, which would be consistent with the idea that the frontal eye field might be the cause of the effects in V4. If you look at coherent interactions between the frontal eye fields and V4, that's shown here.
32:10
And you can see that with attention, there's a lot-- the firing rates between the frontal eye field and V4 become a lot more coherent in the gamma range. There's a big enhancement of coherence with attention, suggesting more effective communication between the two areas. And finally, if we deactivate the frontal eye fields, that's shown here. On the left is the firing rates to the attended and unattended stimulus in the normal condition. And then on the right is the effects
32:42
after we've deactivated the frontal eye field. And you can see that when you deactivate the frontal eye fields, there's no more enhancement of the firing rates in V4 with attention. So the frontal eye fields was necessary for these effects in V4 is the positive experiment. So that's very supportive of the idea that the frontal eye fields and through funnels feedback from that is controlling the activity in V4 with attention.
33:12
OK. So now I'm going to switch over to talking about attention to features of objects rather than to objects at spatial locations. If I show you this classroom scene and I ask you to find the girl in the pink shirt, I assume you can all do that pretty quickly.
33:36
And you must have used your knowledge of what girls in pink shirts look like to guide your attention so quickly to the right object in the scene. You did not scan it pixel by pixel until you came across the object I mentioned. But when you think about it, how this happens seems a little mysterious because for spatially directed attention, if I tell you to attend to the object in the upper right-hand corner of your screen versus the upper left-hand corner
34:06
of your screen, that's not so mysterious because we have lots of maps of space in our visual system and prefrontal cortex and so on. And so we have all kinds of retinotopics, visuotopic maps and so on. But when I ask you to find the girl in the pink shirt, I mean, where is the map of girls, and where is the map of pink shirts? Where's that come from? And if you think about it, it's a very comparable problem to memory recall. So if I ask you to pull up in your mind an image of what
34:44
your mother looks like, I assume most of you can do that, although it turns out there's a significant percentage of people who claim that they cannot visualize anything in their mind, that if you ask them to visualize what their mother looks like, they cannot. But in any case, I assume that most of you can. And it's the same thing. We know from brain imaging experiments that when we ask people to imagine visual things, their visual system gets activated as though it's being used.
35:16
This is kind of a buffer to hold those images and information. So where does it come from? OK. And we did an experiment in humans using magnetoencephalography that identified a region in the prefrontal cortex that seemed to be an important source of this kind of feedback. I'm not going to-- again, there's not going to be time to talk about this particular experiment. But the conclusion that has been published a number of years ago-- but the conclusion of this experiment--
35:51
well, let me just tell you what the experiment was and then what the conclusion was. The experiment was that we had subjects attending to either faces or houses. And we know that faces and houses activate the fusiform face area or the parahippocampal place area. And we had them present at the same spatial location. So we could have used spatial attention to distinguish between one and the other. We had to use attention to objects and their features.
36:20
And what we found was that the face area and the place area went into synchrony with the same region in an area of human prefrontal cortex known as the inferior frontal junction. And experiments from other labs have also suggested that this region is important to the control of attention. But here, it seemed like it had a parallel role to the frontal eye fields with respect to these visual areas that we had found in monkeys, where the frontal eye fields would go into synchronous activity
36:55
with these retinotopic areas. And in this case, the IFJ went into synchrony with either the face area or the place area, depending on what you were attending to. So that could be a source. And so we decided we would test for a role of a structure like this in monkeys. And the particular structure that we focused on, we referred to as the ventral prearcuate region for a number of reasons, including its anatomy and properties and so on. But it's sort of ventral and anterior to the frontal eye
37:31
fields in monkey prefrontal cortex. And we recorded simultaneously on the frontal eye fields from VPA. Also, we did some recordings in other parts of prefrontal cortex and [INAUDIBLE] all during the time when the animal was doing a task in which we could look at the effects of both spatial-based attention and feature-based attention. Now, what was the task? The task was a visual search task, just like in that task where I asked you to find the girl with the green shirt.
38:05
Well, we essentially gave that task to the monkey. The money would start off the trial fixating. And then we would show the monkey a cue-- in this case, a shoe. And that monkey would know from training that that would become the target in the trial, and the animal should search for something that matched that cue. So after a blank delay period in research trials, an array of complex stimuli would appear. Just like you moved your eyes around that picture
38:37
of a classroom, monkeys can move its around eyes the array. But when it got to a stimulus that matched the cue-- we call that the target-- it just holds fixation there. And then it gets reviewed. It found the object. And we had some detection trials where there's just a single object as a neutral. All right. That clear to everybody? OK. But how do we separate out the effects of feature- and spatial-based attention in this task? Well, for spacial-based attention, it's kind of simple.
39:16
We look at the response of a cell to a stimulus in its receptive field-- in one case, where the animal is about to make a saccade to the stimulus. So we know we pay attention to the things we're about to look at. And so that's the attended condition. And then we compare that to the response to that same stimulus when the animal is about to make a saccade someplace else. So it's the unattended condition for spacial attention. But for feature-based attention, we
39:48
have to be a little more clever than that. And what we do is we only look at times when the animal's about to look at something that's not in the receptive field. So spatially directed attention is directed outside the field, as you see up top. And then we compare epochs, time epochs, where the stimulus in the receptive field is the thing that the animal is looking for. So let's say a trial where the animal's been cued to look for a shoe.
40:21
And we compare that response to the same object, the shoe, but when the animal's looking for something else, like a banana or a face. So he's not about to look at it. So spatial attention isn't there. However, feature-based attention is distributed across the whole display. And the animal's going to use that to guide its attention to the target. And we can look at the effects of feature-based attention on that target response. Just in mapping, you can map the receptive fields
40:55
of different areas, including the frontal eye fields and VPA and IT cortex and so on. And what we found was-- and down below, we simply looked at the magnitude of response as you move out from the receptive field center. And it gives us an idea of the receptive field size in the different areas. And what you see is that across these areas, the smallest receptive fields are the frontal eye fields. But in the VPA, the average receptive fields are not that different.
41:32
But we do find in VPA that there's a mixture of receptive fields where some of them look very much like those in the frontal eye fields. And others are big, and they look more like IT cortex. OK, so there's a variety in VPA. And then VPS is just another part of prefrontal that we're not going to talk about. Now, in addition to having receptive fields, VPA also shows object selectivity. The frontal eye fields doesn't. The frontal eye fields don't care what the stimulus is.
42:03
They only care if you're going to move your eyes there. But if you look at anterior IT cortex and you just order responses from the best stimulus to the worst stimulus for every cell, you see this fall off in response, with one selectivity. In VPA, it's not too bad. It's not quite as good as IT cortex, but the cells are pretty selective. And VPS is just another, again, controller. So the cells combine spatial information with receptive fields and object information, as well.
42:35
And now we can look at the response to the cue stimulus at the start of the trial and look at, what's the difference in response to the best stimulus for the cell compared to the worst stimulus for the cell? And we call that the preferred cue for the cell or the non-preferred cue. So let's say a cell might give its best response to a shoe and its worst response to a face, so the shoe is the preferred stimulus. And if you plot that difference for anterior IT cortex--
43:04
that's shown here-- and time 0 is when the cue appears, then you can see that there's a strong response to the preferred cue and no response to the non-preferred cue, so very strong selectivity. In VPA, you also see very good selectivity-- not quite as strong as in IT cortex, but still pretty good selectivity. And VPS, again, is just another control region. So there's stimulus selectivity. Now let's look what happens during the delay period and then at the time when the array comes on.
43:41
In anterior IT cortex, there's not much happening during the delay. And there's a tiny effect when the array comes on of whether the animal-- the firing rate's just a bit higher if the animal's being shown the preferred cue for the cell or the non-preferred. But in VPA, there's a lot going on. During the delay activity, the activity is higher if the animal has just seen the preferred stimulus for those cells compared to the non-preferred. And when the array comes on at time 0,
44:13
you can see the activity in response to the array just goes up, higher than if the animal-- when the animal's seeing the preferred cue to when they see the non-preferred cue. So even at just the start of the array response, the cells are showing higher firing rates if the animal's looking for the stimulus that happens to be their preferred stimulus. They're trying to find that stimulus. OK. And now finally, let's look at the response to the array according to what the animal is looking for.
44:46
And now we can see-- and it's a little complicated. I'll walk you through this. We show these responses for the frontal eye fields and VPA. In the frontal eye fields, the difference in response to the green line and the red line show the effects of spatially directed attention. So the red line's when the animal's looking someplace else. The green line, when it's looking into the receptive field. And so this enhancement is just the enhancement due to spatially directed attention.
45:15
And the difference between the red line and the blue line shows the enhancement for feature-directed attention. So in other words, the stimulus in the receptive field is the stimulus that the animal is looking for on that trial. And then you get this enhancement. And other people have shown this, as well, that if the thing in your FEF receptive field is the target for the animal on that trial, you get this enhancement. And in VPA, we see something very similar.
45:46
So we see an enhancement for spatial attention. We see an enhancement for feature-based attention like in FEF. But the latencies look a little different. So the latency for spatially directed attention seems a little faster in the VPA and for feature-based attention-- sorry. For feature-based attention, seems faster. And for spatial-based attention, it seems slower. And that's shown more clearly in the next slide, where we look at cumulative latency distributions.
46:23
And we can view the latency on a cell by cell basis. So for feature-directed attention, the earliest latencies are found in VPA and then later in the frontal eye fields and even later in places like IT cortex. But for spatially directed attention, the earliest latencies are found on the frontal eye fields. And then they're later in VPA and then even later in areas like IT cortex. So it seems like FEF is finding the stimulus that you're attending to spatially first.
47:00
And maybe VPA is finding the stimulus you're attending to based on features first. Is this consistent with that idea? But to test that, we want to do a causal experiment to see if VPA explains some special role in feature-based attention. So what we did was we injected muscimol and GABA agonist into VPA to deactivate activity in one hemisphere. And then first of all, we measured the animal's performance in the visual search task. And we measured it both pre-muscimol
47:38
when the animal was normal and post-muscimol when the animal has deactivated VPA on one side. Now, the ipsilateral field is unaffected by the VPA deactivation. There's no difference. But in the contralateral field, there's a big increase in errors if the animal's head is VPA deactivated. And there's an intermediate effect if the stimulus is presented on the midline. OK, so at least behaviorally, VPA seems to be important for feature-guided attention and visual research.
48:14
AUDIENCE: On the graph that was presented for the visual search study and you measured latency for each of these-- ROBERT DESIMONE: Each of the cells. AUDIENCE: Right. And was there a conforming effect of saliency there? Because I mean, if you're looking for a banana and everything sounded like red objects, you don't need that to be the preferred item.
48:37
There would be some contextual modulation there. ROBERT DESIMONE: Yeah. So you're saying suppose we did an experiment where let's say we didn't give the animal a cue, and we just looked at the animal's-- just looked at responses as the animal looks around the display. Maybe bananas are more interesting. So it tends to look at bananas more or something like that. I mean, we didn't do that experiment. But I anticipate that if the banana is attracting
49:09
the animal's attention, then you would at least get an enhancement due to spatially directed attention because the animal-- it would be the target of an eye movement. If the animal has no instruction to search for a banana, then I wouldn't expect any effective feature-guided attention because there's no guidance. There's no internal goal that you're trying to satisfy. So I wouldn't expect that. But we didn't actually test that. But we did test-- all of these different objects were randomized,
49:40
and the cue could be anything on any trial. And so it could be a shoe or whatever. And whatever it was, you got these effects. We didn't see that it was preferential for bananas or faces or anything like that. AUDIENCE: OK, because, I mean, it could be possible that the contextual effects are a bit slower than the feed forward-- I mean, whatever you record, so-- you might see that, after a bit of time, you see the response going down. But that's not because the attention [INAUDIBLE]
50:16
moving before. It's just because of lateral effects from some nearby objects. I'm not sure. ROBERT DESIMONE: I'm having a hard time following that. So you're talking about the contextual effects. So there could, in principle, be contextual effects of the objects around the receptive field. Is that what you're saying? AUDIENCE: Yeah, yeah. ROBERT DESIMONE: Yeah. They're random. The location of every given object is randomized in every trial. So the results have been averaged
50:45
over every possible combination of objects and locations. They're not seeing the same array every time. They can't make any prediction about where a stimulus is going to be from one trial to the next. But it's at least-- yeah, so if you do a more complicated analysis-- and we do this, and you can look at this in the published paper. You can see things like if an animal is going to make his eye movement to a stimulus and then another stimulus, you begin to see a little bit of an enhancement
51:19
of the next stimulus after the current one. So you start to see that there's some bridging activity that's occurring across objects in the trial. So it's not just limited to one stimulus at one time. There are bridging effects that do occur across time. And we haven't done this, but we've talked a lot about doing priming experiments and so on. What happens when the stimulus that's the target on one trial is the distractor for another trial?
51:53
The animal does tend to look at those a little bit more. So if it just looked for a shoe on one trial, on the next trial, when it looks at distractors, it's a little bit more likely to look at a shoe because it's been relevant for it. And you do see an effect on firing rates in those cases. AUDIENCE: [INAUDIBLE] yeah. I'm sorry, one more question. ROBERT DESIMONE: Yeah. AUDIENCE: One of the previous slides, like maybe 10 slides back or something, you showed the decoding accuracy
52:20
rate after presentation of a single object versus-- ROBERT DESIMONE: That's way back. AUDIENCE: Yeah, yeah. It would be after-- and you showed-- yeah, right. So I mean, am I interpreting this right? So you see, this has been really one object that's presented to the animal. ROBERT DESIMONE: Yeah. So it's one object presented at a time. Correct. AUDIENCE: Right. But then the next slide, you showed multiple objects being presented. And so the decoding accuracy of the preferred object
52:50
or the [INAUDIBLE] is pretty much the same in both these cases. Was that an artifact? ROBERT DESIMONE: The decoded is what? What's the same in both cases? AUDIENCE: The blue line in this chart. ROBERT DESIMONE: Oh, yeah, that's just there for comparison. AUDIENCE: Ah, I see. ROBERT DESIMONE: Yeah, yeah. On this slide, we're only showing the results from the-- I mean, the only relevant thing are the red and the green line. But the blue line there is just to show you by comparison
53:17
what response would you have-- what decoding would you have had if there had only been that one object. AUDIENCE: Oh, OK. Makes so much sense. It's OK. Thank you. ROBERT DESIMONE: Sorry. Sorry for that confusion. AUDIENCE: Thanks. ROBERT DESIMONE: Yeah. One thing I didn't mention in this experiment was that you kind of use the responses to read the monkey's mind. I think I mentioned that sometimes an unattended stimulus will change color. And if the animal makes a saccade to that,
53:51
then that suggests the animal's not following our instructions. But what we find is that if an unattended object changes color, the decoding performance for the attended object suddenly drops down. And the decoding performance for the unattended object suddenly jumps up. And about within a couple hundred milliseconds, they switch again. So it's as though, for a couple hundred milliseconds, the animal's attention was actually attracted to the color change of the distracting objects
54:27
and then switches back to the object that it was instructed to attend to. And so you can actually see that in the decoding performance when the animal switches his attention around. I don't have that slide here, but that was a finding in that study. OK. So what happens when you do the deactivation? And now we're recording the activity of cells in the frontal eye fields. OK. So on the left shows the results and recordings in the frontal eye fields when the animal's normal
54:58
and on the right after we deactivate VPA. So on the left, you see this enhancement for spatial- and feature-directed attention, like I just showed you. OK. But after we deactivate VPA, the enhancement for feature-based attention just goes away. There's no longer any enhancement when the animal's looking for the stimulus that happens to be in the receptive field. So the cells in the FEF simply don't care anymore if the stimulus in the receptive field is the one the animal's looking for.
55:31
The only thing they care about is the thing that you're going to make the eye movement to. So it suggests that VPA is the source of the critical feature information that enhances the responses of cells in the frontal eye fields and, we believe, lead to the eye movement. So it would take an interaction between these two structures to both find the object and then move your eyes to it. That's how we think it works, across FEF and VPA. Now, but what about the object recognition system?
56:10
So far, we've kind of ignored that here. And is VPA the source of that modulation? And so we repeated this exact same experiment. But instead of recording in the frontal eye fields, we recorded in area V4. And on the left shows the responses in V4 before the deactivation. And now it was the difference in response between the red and the blue line shows you the enhancement for feature-based attention in V4 in normal case. And me and other people have seen this in many, many cases.
56:50
But on the right, you see what happens in V4 after the VPA deactivation. And you can see that feature-based enhancement completely goes away. And on the next slide, we tried as a controlled experiment the same thing but with the spacial-based attention. So here, on the left is the enhancement in V4 for just spatially attending to the stimulus in the receptive field. And then after VPA deactivation, it doesn't go away. It's basically statistically not different.
57:29
So VPA was not necessary to get the enhancement due to spatially directed attention in V4.
57:39
However, now if we compare the effects of FEF deactivation in V4, now we find that if you look at the effects of feature-based attention, feature-based attention in V4-- this is in the normal case, and this is after VPA deactivation, which remains. So the FEF deactivation had no effect on feature-based enhancement in V4. Incidentally, the people in my lab were completely split in predicting how this experiment would turn out because there are some theories of feature-based attention
58:19
that propose that feature-based attention in the end always relies on a spatial-based attention, that you find the object based on features and then you use your spatial attention system to modulate your activity. And so in that case, the FEF deactivation would have eliminated feature-based attention in V4. But it did not. So this is a more simple-minded case. So FEF is not important for feature-based attention effects in V4. However, if you look at the effects
58:52
of spatial-based attention in the normal case, there is an enhancement in the normal condition. But after you deactivate the FEF, that enhancement is gone. So the FEF and the VPA have completely opposite effects on the firing rates in V4 with attention. So it's really a double dissociation. Here's sort of a cartoon representation of that finding. So imagine this is a display in the visual field where you're searching for something. And you're about to make an eye movement to something.
59:35
And then this represents a map in VPA where the intensity represents the level of activity at that location. So what we imagine in VPA, the level of activity marks the potential targets. So the biggest enhancement is going to be at the location where you have an object that matches what you're looking for. But it'll fall off according to object similarity. But in FEF, ultimately, you have the representation of where the eyes are going to move. It's the saccade target.
1:00:10
And it's the activity that's going from VPA to FEF that allows that saccade target to remain. And that, we believe, is what generates the eye movement. So you need this, what we believe is an integrated cortical system between FEF and VPA, which is actually adjacent to it. These are adjacent structures that we believe are mediating these effects in the feature-guided search, which is the most common situation in which we're directing attention.
1:00:51
Now, I acted as though VPA has these direct feedback connections to all these visual areas and that we were observing the effects of that influence. But actually, when you looked at the anatomical literature, it was not at all clear that VPA had direct anatomical connections to these visual areas. It has indirect, for sure. That's been shown in some studies. You can go from VPA through other structures to get to them, but not direct. FEF, on the other hand, it's quite clear it
1:01:29
has direct connections. And so we wondered, how could we actually sort this out? How do we determine this for sure? And so the last thing I was just going to talk about is our studies of the connectome of the prefrontal cortex. So we decided that we couldn't just rely on single anatomical tracer injections in animals because there's just too much variability, and it's almost impossible to really compare one animal to the next when we only have a single injection site, at most two.
1:02:04
So we really need a much denser anatomical map. And maybe someday we'll be able to do this all with molecular barcoding or something. But for now, to get a dense map of the prefrontal cortex, we decided to use electrical stimulation with fMRI. And this is a study that was done by Rui Xu in the lab where, here, you can see prefrontal cortex in two monkeys. And all these blue dots are sites where we electrically stimulated. And then it's known from the work
1:02:38
of Nikos Logothetis and others that if you electrically stimulate a site and then have the animal in an MRI machine and can measure the BOLD signal, then you will get an increase in the BOLD signal at monosynaptically connected sites. And so this is the natural view of the lateral prefrontal cortex. And this is a blown-up flattened map of the prefrontal cortex. And so for some of the stimulation sites, we stimulated at different depths, like down the bowl sulcus and things like that.
1:03:15
So in two animals, we created this dense map of stimulation sites. These had to be collected over many, many different recording sessions because we could only do a small number of simulations in one given MRI session. But we collected these over time. And that gave us the connectome of the lateral prefrontal cortex. And it's summarized in one form here. So in these boxes are these flattened maps of the prefrontal cortex. And in the center is a flattened map of the whole rest of the cortex.
1:03:54
So this is the orbitofrontal cortex. This is the lateral prefrontal cortex. This is the dorsomedial prefrontal, the posterior medial cortex, the parietal cortex, the temporal cortex, and the visual cortex.
1:04:10
And you can see it's all-- so you can see that there's colored-in regions in the rest of the cortex. And this is the sum total of activated cortex from our stimulation sites. So you can see that these sites include most of the association cortex. It excludes primary sensory areas, primary motor areas, but includes most of the rest of the cortex. And furthermore, we found that there's an organization to each one of these-- we call them domains. There's six major domains of connections.
1:04:55
The temporal cortex and the parietal cortex, for example, are two domains. And they're color coded here so that-- let's see. So here's the color-coded stimulation sites in the prefrontal cortex compared to the colored cortex activation zones in the temporal cortex. And so you can see that as you go from the green to the red stimulation sites in prefrontal, you can move from the green to the red activation zones in the temporal cortex. And then if you move along a different axis,
1:05:30
say, from green to blue, you move along a different axis in the temporal cortex, actually, from green into blue going up through the dorsal flank of the sulcus. And in fact, if you do a principal component analysis of all of the patterns that you see in the different domains, you get that there's a couple principal components that can describe the overall organization of the prefrontal to domain mapping. Now, it's different for the different mappings. But each one has a mapping.
1:06:08
And they're at least partially overlapping. Between the temporal and the parietal cortex is a lot of overlap. So it's almost the same sites in the temporal cortex and the parietal cortex. It's just that the axes are oriented differently. And so that means for the temporal and parietal cortex, the prefrontal cortex takes those two domains, overlaps them, and then both integrates them and feeds back to comparable levels in each processing system. I mean, another way of looking at this
1:06:53
is that the prefrontal cortex contains maps of all of the major processing domains in the rest of the brain.
1:07:04
In some ways, it's kind of analogous to retinotopic maps. But here, it's not mapping the retina. It's mapping the brain in these overlapping maps. And we illustrated this in a kind of cartoon fashion here where you can imagine these-- if you take grids to representing the spatial layout of the different processing domains, and then how are they mapped in prefrontal cortex? You will end up with, at least partially-- in some cases, largely-- overlapping grid representations in
1:07:38
the lateral prefrontal cortex. So again, it's integrating these across all the major different processing domains. So certainly the role of prefrontal cortex is a lot broader than just enhancing processing according to what you're attending to. It's really coordinating activities across all the major processing systems in a domain, of which this attentional system is just one part. And to go back to the-- actually, I should have pointed this out. To go back to this original question of,
1:08:07
does VPA have direct connections to V4, at least as measured this way? The answer is no. It's got to be going through other connections, which could be through the inferior temporal cortex, for example. We just don't know yet. But they're not direct. But in the course of doing that experiment, we discovered something that we think is actually much more interesting about the general organization of the connections of prefrontal cortex to the rest of the brain. AUDIENCE: Professor.
1:08:35
I'm just curious. I'm not very sure I understand how you get the maps in the last slides. Is that true, you're stimulating once the actual and then map to locations [INAUDIBLE] the whole brain? ROBERT DESIMONE: So you stimulate one side. So let's say we stimulate this site here in the prefrontal cortex and then the temporal cortex that maps to some portion of this green zone that you'll see here, right? It's hard to see all the different color shades. In the paper, we're actually coming up
1:09:09
with different ways of illustrating this. But yeah, so again, these are color gradients. So as you go from green-- through the different shades of green through the different shades of red-- in the temporal cortex, you go from the different shades of greens into the different shades of red. AUDIENCE: And the color is representing the coordinates in that-- ROBERT DESIMONE: Yes. It's representing the coordinates in the connected zone. So this site here in the-- this is actually in the frontal eye
1:09:43
field-- will be represented in the temporal cortex. But it will also be represented in the parietal cortex. So you can see, here's the green sites over here. So the sites will map to this part of temporal cortex and to this part of parietal cortex. So as you go from green to red in the prefrontal cortex, you go from posterior to anterior in temporal cortex. And you go from posterior to anterior in the intraparietal sulcus of the parietal cortex. So you can see this is a--
1:10:21
we say this is a trajectory that is mapped in the temporal cortex and in the parietal cortex. AUDIENCE: Thank you. ROBERT DESIMONE: Yeah. So you could see where the parallel areas were. So the region that connects to the same prefrontal sites and presumably has something related functions are the most anterior part of the parietal processing stream and the most anterior part of the temporal processing stream. AUDIENCE: So it's kind of like a retinotopic map,
1:10:52
but the retinotopic is mapped by the prefrontal cortex circuits. ROBERT DESIMONE: That's right. That's right. And the principal component analysis, it's as though it discovered the horizontal and vertical axes of the visual field. And in this case, it's the horizontal and vertical axes of the cortex. OK, so we're at the end of our time, and we're at the end of my slides.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat