Skip to main content
Loading Explore guides…
Neural Networks vs Brain Neurons · Acedra
Home
Explore
Computer Science
Neural Networks vs Brain Neurons
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
54 segments
Copy
Hide video
0:00
The year 2022, which has recently passed, can easily be called the year of neural networks. After all, we got a language model which can write you a surprisingly decent essay, an AI art generator, and even a platform which turns any image into Anime. All of this was made possible by the development of artificial neural networks, a type of computing system modeled as a network of interconnected nodes which can learn to solve problems by recognizing patterns hidden in the training data.
0:35
And if you go online and google "<i>artificial neural networks</i>", you are likely to see statements such as they work pretty much like your brain. Well, you see, such claims, although attractive, can be a bit misleading because biological neurons are actually much more powerful than was previously thought. In this video, we will see why individual neurons in your brain function essentially like full blown neural networks themselves, equipped with insane information processing capabilities,
1:06
as well as some of the physiological mechanisms that account for this computational complexity. If you're interested, stay tuned.
1:21
Before we begin, I'd like to warn you that this video is not on artificial neural networks per se. We're not going to talk about back-propagation or gradient descent or any of that stuff in detail but we are going to talk about their relationship with biological neurons, as well as some of the applications to modern neuroscience. But more on that later. Early neural networks were indeed inspired by the descriptions of biological neurons, which people thought to be accurate at the time.
1:50
In fact, the birth of machine learning as we know it, can be traced back to 1943 when Walter Pitts and Warren McCulloch introduced the notion of a perceptron. Despite the fancy name, the idea is quite simple. The perceptron was created to function like an individual nerve cell, which in this doctrine, works like a simple summator and comparator. This just means that it receives an input set of numbers, multiplies them by some coefficients, also called weights,
2:20
sums everything together and compares the result with a threshold. If the resulting value exceeds the threshold, the perceptron sends a number <b>1</b> as an output to its neighbors. You interconnect a bunch of these perceptrons with each other such that the output of 1 serves as an input to a downstream perceptron, encode an input and output layer and boom! You've got a neural network! And to train the network means to somehow adjust the values of these input weights
2:49
to make it match an input to a correct output. But at that point, the fields of machine learning and neurobiology pretty much diverge. Over the years, people have invented a bunch of activation functions, organized neurons in a multitude of different network architectures, came up with algorithms to change the weights efficiently, and so much more. But because we still refer to the nodes in such networks as neurons, a lot of people believe that biological neurons in the brain
3:20
function exactly like their perceptron counterparts. The main goal of this video is to exonerate biological neurons and show that single cells are much more computationally powerful and sophisticated than you might think. But to understand the computational complexity of a neuron, it's helpful to remind ourselves of the basic biology behind neural computations. If you open any neuroscience textbook, one of the first things you'll see is the structure of a typical neuron
3:49
usually consisting of dendrites, soma or cell body, and an axon. Let's put dendrites aside from now as they will become key players later on.
4:00
As you probably know, the key property of neurons is that they are electrically excitable cells which means they have the capacity to generate brief electrical pulses that are propagated to other neurons, forming a basis of communication between cells. In biological systems, electric charge is carried by ions such as sodium, potassium, chloride and calcium which are floating both inside and outside the cells in different proportions.
4:30
Cells are separated from the outside world by a lipid membrane, a barrier normally impermeable to ions. However, neurons possess special proteins forming channels through which specific ions can cross the membrane and which can open and close through a variety of mechanisms, as we'll see further. So by regulating the flow of ions through the channels, cells can control the balance of electric charges and thus, control the <i><u>membrane voltage</i></u>.
5:00
Namely, when positive ions flow into the cell, they are said to depolarize the membrane, increasing the voltage, making the potential more positive, and vice versa for negative ions. From the whole zoo of ion channels, we'll be mostly interested in what's called voltage gated channels, which can open and close depending on the value of membrane potential.
5:26
In school, we usually learn about voltage gated channels in the context of action potential generation. So just to remind ourselves, an action potential, which is the unit of communication, is often said to be all or none output of a neuron. In a simplified description, it is generated at the origin of an axon where there is a large number of special sodium channels that open when the membrane voltage exceeds a certain threshold. Because sodium is a positively charged ion,
5:56
when it rushes into the cell, membrane gets depolarized even further, recruiting more sodium channels to open and so forth, like a positive feedback loop. That wave of sodium channels opening up is transmitted along the axon and gets indirectly passed onto other neurons by the means of synaptic transmission. However, sodium channels don't stay open forever and eventually they close while another type of channels opens, allowing potassium, also a positively charged ion, to leave the cell
6:30
bringing the membrane voltage back to its resting level. <b>Tada!</b> We have just generated an action potential and sent one bit of information to downstream neurons. You might already see why the initial description of a neuron as a perceptron seemed reasonable. After all, we have just witnessed how a thresholding function is implemented in a neuron through voltage gated channels. So that part is accurate, right?
6:57
Indeed, the main problem with the perceptron is mostly with inputs, not the output. It's time for us to get back to the dendrites we've been putting off and see how exactly they contribute to the computations on incoming information. Now, the way it is often presented, and this is the way I was taught, is that dendrites function like passive receivers of information. You see, the signal is transmitted between the two cells cells when the axon of one neuron forms a special connection, a synapse,
7:27
with the dendrite of another neuron. Upon the action potential of a sender neuron, or pre-synaptic cell, there is a chain of chemical reactions which leads to a release of signal molecules and eventually resulting in the opening of neurotransmitter gated ion channels on the membrane of the receiver post-synaptic neuron. This flux of ions across the membrane of a post-synaptic cell leads to its depolarization, which is propagated along the dendritic tree
7:56
and the laws of such propagation of electrical signals in passive membrane is conventionally described by a <i>Cable theory</i>. In that paradigm, the dendrites are just that. Cables, whose only job is to transmit electrical signals to the soma. For those of you who are more interested, dendrites are actually treated as imperfect leaky cables, since a single portion of a dendrite is described as a patch of lipid membrane with capacitive properties
8:26
and a resistive component corresponding to a leak of ions through passive, permanently open ion channels. As a result, the depolarization level is attenuated as it travels along the dendrite. In other words, the function of dendrites is reduced to summing the incoming signals where the weight of a synaptic input is determined by the amount of receptors and how far away the synapse is from the cell body. Together with the threshold happening at the soma,
8:59
this gives us the biological basis for the perceptron model. However, as neuroscientists study the cells more closely, it became clear that dendrites are anything but passive cables and the reason for that is that, apart from permanently open passive ion channels which give them their leaky properties, dendrites are also covered by a myriad of voltage gated ion channels which equip the dendritic tree with powerful information processing capabilities even more powerful than the soma itself.
9:31
For example, dendrites actually contain voltage gated sodium channels similar to the ones found on the axon They allow action potentials to travel in reverse direction and influence the post-synaptic sites. This is called backpropagation and it plays an important role in synaptic plasticity, adjusting the weights of inputs. Fast sodium channels also give dendrites an ability to generate their own small action potential like depolarizations, which can transiently amplify synaptic inputs.
10:08
Another important type of channel is <u><b>NMDA receptor</u></b>, which requires both sufficient membrane depolarization and the presence of neurotransmitter to open, thus functioning like a kind of coincidence detector. NMDA channel is non-selective to cations, allowing both calcium and sodium to flow into the cells which has great implications for things like synaptic plasticity.
10:35
This depolarizing event is often called an NMDA spike. Unlike pure sodium spike plants, these are generated by the influx of calcium as well and are characterized by a much longer time scale of hundreds of milliseconds. Importantly to our discussion, NMDA channels allow the dendrites to perform non-linear integration of incoming information, This equips dendrites with vast computational complexity. For example, dendrites can discriminate the order of incoming action potentials.
11:10
This means that if you took a group of synapses on a dendrite of a <b><u>cortical pyramidal</b></u> neuron then a sequential activation of them in one direction will produce a fundamentally different electrical and chemical response than activation in reverse direction. And this is sensitive not only to order, but also to the velocity of activation. Thus, single neurons have a mechanism to process temporal patterns and generate sequence selective output.
11:40
NMDA spikes have also been shown to enhance stimulus selectivity in visual cortex of awake animals, thus contributing to behaviorally relevant neural computations in the intact brain. If you're interested in learning more about this paper and the key result, I have a dedicated short clip available for my Patreon supporters. If you would like to support the channel, vote for video topics and enjoy the bonus content, you can find more information by following the link in the description.
12:09
But today I'd like to focus on something different. What if I told you that certain neurons in the human cortex are capable of performing a type of computation that was previously thought to require a multi-layered neural network to implement? Indeed, in 2020, a group of researchers from the laboratory led by <i>Matthew Larkum</i> published a paper titled "<i>Dendritic action potentials and computation in human layer 2/3 cortical neurons</i>"
12:38
where they demonstrated a unique property of human pyramidal neurons. More specifically, they performed simultaneous recordings of electrical activity at the cell body and a dendrite and found a new type of electrical response that was initiated at the dendrites by a sufficiently strong excitatory input. This waveform was termed as dendritic calcium action potential. As the name suggests, such electrical events are caused by the influx of calcium ions,
13:10
but have a shorter time scale compared to NMDA spikes. Remarkably, these calcium spikes, which have not been described in other mammalian species, are highly selective to a particular input strength. This means that if you stimulate a neuron with too weak of a current, the membrane voltage would stay below the threshold for opening of calcium channels, so no dendritic spikes would be observed. If, however, you increase the current too much, no spikes would be observed either.
13:45
You would need to provide the dendrite with just the right strength of stimulation to trigger them. This fact may not seem like a big deal at first, but it's really important for the way we should treat biological neurons. To better understand what it really means for the neural computations, let's talk about logical operations. As you probably know, the device you're watching this on stores the information in bits. binary units with only two possible states, zero and one. Oh, and by the way,
14:16
I will use terms zero and one and false and true interchangeably throughout the rest of the video. To perform computations on binary data, computers use what are called <i>bitwise operations</i>, which are like addition and multiplication, but in binary world. They are performed by logic gates, which are the building blocks of all the digital hardware. Individual logic gates perform simple Boolean operations which you have probably heard about,
14:41
with the two most popular ones being AND and OR operations. For example, the AND gate receives two inputs and outputs one if and only if both of the inputs are one, while the OR gate outputs one when at least one of the inputs is equal to 1. This is also known as inclusive OR, because the output is true when both of the inputs are true as well. So if we view the inputs as Venn diagrams, the intersection is included.
15:15
There is another useful operation which is called exclusive OR, or XOR for short. As the name suggests, the XOR gate outputs true when exactly one of the inputs is true, but not both.
15:30
By the way, notice that this agrees with our everyday interpretation of the word '<i>OR</i>' when we say things like "<i>Would you like a cup of coffee or tea?</i>" the word "OR" is usually understood in the exclusive sense and in computers XOR can be used for things like comparing two numbers. Importantly to our today's discussion, XOR is what's called a linearly non separable function, and this simply means that there is no line for two dimensions, plane for three dimensions,
16:01
or hyperplane for n dimensions that would separate different classes of output. For example, let's consider a perceptron which receives just two inputs. Similarly to the logic gate, call them x & y. As we have discussed earlier, all perceptron does is multiply the inputs by their weights, adds everything and compares the result with a threshold. If we denote the weights as A and B, then the perceptron essentially solves the inequality <i>ax</i> plus<i> by</i> is bigger or equal than threshold.
16:37
But if you consider the geometric representation of this inequality, it's easy to see that this is essentially a line separating the two halves of the xy plane.
16:49
Similarly, if perceptron had three inputs, this equation would correspond to a plane cutting the 3D space into two halves and so forth. That's why a single perceptron can function as a classifier when two output classes are located on opposite sides of that line. In other words, when they are linearly separable, returning back to our logic gates, the inputs would be limited to zero and one.
17:18
If we visualize the AND gate, it's easy to see that there is a line separating the true and false outputs, so a perceptron can act as an AND gate, and it is possible to turn it into an OR gate just by lowering the threshold. The XOR gate, however, is different. Notice that there is no line that would separate the 0 outputs from the 1's. which makes it a linearly non-separable function. This is why to perform the XOR operation, a multi-layered network is required.
17:51
And it was believed to be true for biological neurons as well that individual cells can't compute the XOR function until, of course, that paper came out. Remember that the described dendritic spikes showed a prominent selectivity towards the strength of a stimulus. So, for example, let's say the neuron has two sets of synapses, A & B. When either one of the two sets is activated, the excitation is large enough to elicit a dendritic spike
18:21
which can propagate to the soma and trigger the action potential. However, if both sets of synapses are activated at the same time, the strength of incoming current exceeds the optimal value for dendritic spike generation, and no such event is observed. In other words, the dendrite just performed the XOR operation on A & B inputs. How awesome is that? For those of you who are curious about the biophysical mechanisms of such sensitivity, this largely remains unresolved.
18:52
However, through computer simulations, the authors were able to show that this property of dendritic action potentials can be explained by a combination of known voltage gated calcium channels and special potassium channels that are sensitive to both voltage and calcium concentration.
19:11
But if biological neurons are quite complex computational devices on their own, how does this fit into the picture of existing neural networks that were developed based on oversimplified assumptions? Should we just throw everything away and start fresh? Well, I think quite the contrary. There is a promising future for the synthesis of ideas from modern neuroscience and deep learning. In particular, I would like to conclude this video by discussing a very elegant paper, titled,
19:41
"<i>Single cortical neurons as deep artificial neural networks</i>" The authors asked an interesting question. Can a deep neural network accurately capture the complex information processing of single neurons in our brains? And if so, how should this equivalent network look like? To find an answer, they first created a detailed, biophysically realistic model of a single cortical neuron using a reconstructed morphology and populating it with all sorts of differential equations
20:14
describing the dynamics of membrane voltage, opening and closing of ion channels of different types, fluxes of ions across the membrane, and so forth. This detailed spatial model was bombarded with a set of incoming inputs and the voltage trace at the soma was recorded as the output. The authors then trained a deep convolutional neural network with various number of layers to see if it could learn the complex input-output relationship of the biophysical model
20:45
when the network receives the identical set of synaptic inputs. As a result, they found that it required between 5 and 8 layers to accurately predict output spikes at voltage values of the detailed model.
21:01
Interestingly, removing NMDA channels from the model drastically reduced the complexity of the equivalent network which now could predict the output with just a single hidden layer. This demonstrates the importance of dendritic non-linearities and NMDA channels in particular, for equipping neurons with vast computational complexity. And surprisingly, the deep neural network that was trained purely on randomly scattered synaptic inputs
21:31
was able to generalize and could faithfully predict the output, even for spatially clustered and synchronously activated synapses that it had never encountered before. So in a way, it was able to grasp the underlying biophysics of the which was never explicitly specified. So what does this all tell us? First of all, this study suggests that single cortical neurons with their non-linear or dendritic integration properties are indeed sophisticated computational units on their own.
22:05
And this computation is comparable with a multi-layered convolutional network, which, if you think about it, is a pretty mind-blowing thing for a single cell to do. It also offers a practical advantage to model individual neurons more efficiently, because even the deep neural network with eight layers is 2000 times faster than running the detailed model, which requires solving a myriad of partial differential equations.
22:32
Are you interested in learning more about the topics discussed here, such as neural networks and differential equations, but not sure where to start? Well, in that case, you are definitely going to love our today's sponsor, <b><u>Brilliant.org.</b></u> It is a revolutionary educational platform which allows you to advance in STEM fields in a fun and engaging way. The key feature of Brilliant is that you get to learn by doing through exploring interactive demonstrations
23:00
and solving problems along the way to ensure that you grasp even the most complicated concepts. Brilliant offers thousands of lessons of different levels of difficulty, from math and computer science fundamentals to things like vector calculus and computational biology. And the new ones are being added every month. For example, if you enjoyed this video, then you might be interested in a course on artificial neural networks. It covers a lot of concepts that you have just watched,
23:29
such as the perceptron models, linear separability, and convolutional networks in more depth while also diving into more complex topics like the advanced network architectures. Don't hesitate to try Brilliant and begin growing your knowledge base in bite-sized chunks just by dedicating 15 minutes a day. Follow the link down in the description to get started for free and the first 200 people to use it will also get 20% off Brilliant's premium annual membership. All right, let's recap.
23:58
In this video, we have seen how the presence of voltage gated ion channels turns dendrites from being merely passive conductors of electricity to active computational units. Moreover, individual dendritic branches have the capacity to perform exclusive-or operation on their inputs, a type of computation that was previously thought to require multi-layered networks. And finally, we have seen how the complex input-output transformations of information inside biological neurons
24:29
require a computational power of a whole convolutional deep neural network. So, hopefully, I was able to convince you that even at level of single cells, the brain is incredibly complex and fascinating, and that next time you hear statements that individual neurons essentially function like linear summators, you'll take those claims with a grain of salt. If you liked the video, share it with your friends, subscribe to the channel if you haven't already and press the like button.
24:57
Stay tuned for more interesting topics coming up! Goodbye and thank you for the interest in the brain.
25:11
(Subtitles by Crimson Ghoul)
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat