Skip to main content
Loading Explore guides…
Computer Memory Overview · Acedra
Home
Explore
Computer Science
Computer Memory Overview
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
68 segments
Copy
Hide video
0:00
Have you ever wondered what’s happening inside your computer when you load a program or video game? Well, millions of operations are happening, but perhaps the most common is simply just copying data from a solid-state drive or SSD into dynamic random-access memory or DRAM. An SSD stores all the programs and data for long-term storage, but when your computer wants to use that data, it has to first move the appropriate files into DRAM, which takes time,
0:33
hence the loading bar. Because your CPU works only with data after it’s been moved to DRAM, it’s also called working memory or main memory. The reason why your desktop uses both SSDs and DRAM is because Solid-State Drives permanently store data in massive 3D arrays composed of a trillion or so memory cells, yielding terabytes of storage, whereas DRAM temporarily stores data in 2D arrays composed of billions of tiny capacitor memory cells yielding gigabytes of working memory.
1:09
Accessing any section of cells in the massive SSD array and reading or writing data takes about 50 microseconds whereas reading or writing from any DRAM capacitor memory cell takes about 17 nanoseconds, which is 3000 times faster. For comparison, a supersonic jet going at Mach 3 is around 3000 times faster than a moving tortoise. So, the speed of 17 nanosecond DRAM versus 50 microsecond SSD is like comparing a supersonic jet to a tortoise.
1:45
However, speed is just one factor. DRAM is limited to a 2D array and temporarily stores one bit per memory cell. For example, this stick of DRAM with 8 chips holds 16 gigabytes of data, whereas a solid-state drive of a smaller size can hold 2 terabytes of data, more than 100 times that of DRAM. Additionally, DRAM requires power to continuously store and refresh the data held in its capacitors. Therefore, computers use both SSDs and DRAM and,
2:21
by spending a few seconds of loading time to copy data from the SSD to the DRAM, and then prefetching, which is the process of moving data before it’s needed, your computer can store terabytes of data on the SSD and then access the data from programs that were preemptively copied into the DRAM in a few nanoseconds. For example, many video games have a loading time to start up the game itself, and then a separate loading time to load a save file.
2:52
During the process of loading a save file, all the 3D models, textures, and the environment of your game state are moved from the SSD into DRAM so any of it can be accessed in a few nanoseconds, which is why video games have DRAM capacity requirements. Just imagine, without DRAM, playing a game would be 3,000 times slower. We covered solid-state drives in other videos, so in this video, we’re going to take a deep dive into this 16-gigabyte stick of DRAM. First,
3:26
we’ll see exactly how the CPU communicates and moves data from an SSD to DRAM. Then we’ll open up a DRAM microchip and see how billions of memory cells are organized into banks and how data is written to and read from groups of memory cells. In the process, we’ll dive into the nanoscopic structures inside individual memory cells and see how each capacitor physically stores 1 bit of data. Finally, we’ll explore some breakthroughs and
3:56
optimizations such as the burst buffer and folded DRAM layouts that enable DRAM to move data around at incredible speeds. A few quick notes. First, you can find similar DRAM chips inside GPUs, Smartphones, and many other devices, but with different optimizations. As examples, GPU DRAM or VRAM, located all around the GPU chip, has a larger bandwidth and can read and write simultaneously, but operates at a lower frequency, and DRAM in your smartphone
4:32
is stacked on top of the CPU and is optimized for smaller packaging and lower power consumption. Second, this video is sponsored by Crucial. Although they gave me this stick of DRAM to model and use in the video, the content was independently researched and not influenced by them. Third, there are faster memory structures in your CPU called cache memory and even faster registers. All these types of memory create a
5:01
memory hierarchy, with the main trade-off being speed versus capacity while keeping prices affordable to consumers and optimizing the size of each microchip for manufacturing. Fourth, you can see how much of your DRAM is being utilized by each program by opening your computer’s resource monitor and clicking on memory. Fifth, there are different generations of DRAM, and we’ll explore DDR5. Many of the key concepts
5:29
that we explain apply to prior generations, although the numbers may be different. Sixth, 17 nanoseconds is incredibly fast! Electricity travels at around 1 foot per nanosecond, and 17 nanoseconds is about the time it takes for light to travel across a room. Finally, this video is rather long as it covers a lot of what there is to know around DRAM. We recommend watching it first at one point two five times speed, and then a second time at
6:00
one and a half speed to fully comprehend this complex technology. Stick around because this is going to be an incredibly detailed video. To start, a stick of DRAM is also called a Dual Inline Memory Module or DIMM and there are 8 DRAM chips on this particular DIMM. On the motherboard, there are 4 DRAM slots, and when plugged in, the DRAM is directly connected to the CPU via 2 memory channels that run through the motherboard. Note that the left two DRAM
6:34
slots share these memory channels, and the right two share a separate channel. Let’s move to look inside the CPU at the processor. Along with numerous cores and many other elements, we find the memory controller which manages and communicates with the DRAM. There’s also a separate section for communicating with SSDs plugged into the M2 slots and with SSDs and hard drives plugged into SATA connectors. Using these sections, along with data mapping tables,
7:04
the CPU manages the flow of data from the SSD to DRAM, as well as from DRAM to cache memory for processing by the cores. Let’s move back to see the memory channels. For DDR5 each memory channel is divided into two parts, Channel A and Channel B. These two memory channels A and B independently transfer 32 bits at a time using 32 data wires. Using 21 additional wires each memory channel carries an address specifying where to read or write data and, using
7:42
7 control signal wires, commands are relayed. The addresses and commands are sent to and shared by all 4 chips on the memory channel which work in parallel. However, the 32-bit data lines are divided among the chips and thus each chip only reads or writes 8 bits at a time. Additionally, power for DRAM is supplied by the motherboard and managed by these chips on the stick itself. Next, let’s open and look inside one of these
8:13
DRAM microchips. Inside the exterior packaging, we find an interconnection matrix that connects the ball grid array at the bottom with the die which is the main part of this microchip. This 2 gigabyte DRAM die is organized into 8 bank groups composed of 4 banks each, totaling 32 banks. Within each bank is a massive array, 65,536 memory cells tall by 8192 cells across, essentially rows
8:47
and columns in a grid, with tens of thousands of wires, and supporting circuitry running outside each bank. Instead of looking at this die, we’re going to transition to a functional diagram, and then reorganize the banks and bank groups. In order to access 17 billion memory cells, we need a 31-bit address. 3 bits are used to select the appropriate bank group, then 2 bits to select the bank. Next 16 bits of the address are used to determine the exact row out of 65
9:21
thousand. Because this chip reads or writes 8 bits at a time, the 8192 columns are grouped by 8 memory cells, all read or written at a time, or ‘by 8’, and thus only 10 bits are needed for the column address. One optimization is that this 31-bit address is separated into two parts and sent using only 21 wires. First, the bank group, bank, and row address are sent, and then after that the column address. Next, we’ll look inside these physical memory cells, but first,
9:58
let’s briefly talk about how these structures are manufactured as well as this video’s sponsor. This incredibly complicated die, also called an integrated circuit, is manufactured on 300-millimeter silicon wafers, 2500ish dies at a time. On each die are billions of nanoscopic memory cells that are fabricated using dozens of tools and hundreds of steps in a semiconductor fabrication plant or fab. This one was made by Micron which manufactures around
10:31
a quarter of the world’s DRAM, including both Nvidia’s and AMD’s VRAM in their GPUs Micron also has its own product line of DRAM and SSDs under the brand Crucial which, as mentioned earlier, is the sponsor of this video. In addition to DRAM, Micron is one of the world’s leading suppliers of solid-state drives such as this Crucial P5+ M2 NVME SSD. By installing your operating system and video games on a Crucial NVMe solid-state drive, you’ll be sure to have
11:07
incredibly fast loading times and smooth gameplay, and if you do video editing, make sure all those files are on a fast SSD like this one as well. This is because the main speed bottleneck for loading is predominantly limited by the speed of the SSD or hard drive where the files are stored. For example, this hard drive can only transfer data at around 150 megabytes a second whereas this Crucial NVMe SSD can transfer data at a rate of up to 6,600 megabytes a second, which,
11:42
for comparison is the speed of a moving tortoise versus a galloping horse. By using a Crucial NVMe SSD, loading a video game that requires gigabytes of DRAM is reduced from a minute or more down to a couple seconds. Check out the Crucial NVMe SSDs using the link in the description below.
12:09
Let’s get back to the details of how DRAM works and zoom in to explore a single memory cell situated in a massive array. This memory cell is called a 1T1C cell and is a few dozen nanometers in size. It has two parts, a capacitor to store one bit of data in the form of electrical charges or electrons and a transistor to access and read or write data. The capacitor is shaped like a deep trench dug into silicon and is composed of two conductive surfaces separated by a dielectric
12:44
insulator or barrier just a few atoms thick, which stops the flow of electrons but allows electric fields to pass through. If this capacitor is charged up with electrons to 1 volt, it’s a binary 1, and if no charges are present and it’s at 0 volts, it’s a binary 0, and thus this cell only holds one bit of data. Designs of capacitors are constantly evolving but in this trench capacitor, the depth of the silicon is utilized to allow for larger capacitive storage,
13:17
while taking up as little area as possible. Next let’s look at the access transistor and add in two wires. The wordline wire connects to the gate of the transistor while the bitline wire connects to the other side of the transistor’s channel. Applying a voltage to the wordline turns on the transistor, and, while it’s on, electrons can flow through the channel thus connecting the capacitor to the bitline. This allows us to access and charge up the capacitor
13:47
to write a 1 or discharge the capacitor to write a 0. Additionally, we can read the stored value in the capacitor by measuring the amount of charge. However, when the wordline is off, the transistor is turned off, and the capacitor is isolated from the bitline thus saving the data or charge that was previously written. Note that because this transistor is incredibly small, only a few dozen nanometers wide, electrons slowly leak across the channel, and thus over time the
14:19
capacitor needs to be refreshed to recharge the leaked electrons. We’ll cover exactly how refreshing memory cells works a little later. As mentioned earlier, this 1T1C memory cell is one of 17 billion inside this single die and is organized into massive arrays called banks. So, let’s build a small array for illustrative purposes. In our array, each of the wordlines is connected in rows, and then the bitlines are connected in columns. Wordlines and bitlines
14:53
are on different vertical layers so one can cross over the other, and they never touch.
15:00
Let’s simplify the visual and use symbols for the capacitors and the transistors. Just as before, the wordlines connect to each transistor’s control gate in rows, and then all the bitlines in columns connect to the channel opposite each capacitor. As a result, when a wordline is active, all the capacitors in only that row are connected to their corresponding bitlines, thereby activating all the memory cells in that row. At any given time only one wordline is
15:31
active because, if more than one wordline were active, then multiple capacitors in a column would be connected to the bitline and the data storage functionalities of these capacitors would interfere with one another, making them useless. As mentioned earlier, within a single bank there are 65,536 rows and 8,192 columns and the 31-bit address is used to activate a group of just 8 memory cells. The first 5 bits select the bank, and the next 16-bits are sent to a row decoder
16:06
to activate a single row. For example, this binary number turns on the wordline row 27,524, thus turning on all transistors in that row and connecting the 8,192 capacitors to their bitlines, while at the same time the other 65 thousandish wordlines are all off. Here’s the logic diagram for a simple decoder. The remaining 10 bits of the address are sent to the column multiplexer. This multiplexer takes in the 8192 bitlines on the top, and,
16:42
depending on the 10-bit address, connects a specific group of 8 bitlines to the 8 input and output IO wires at the bottom. For example, if the 10-bit address we this, then only the bitlines 4,784 through 4,791 would be connected to the IO wires, and the rest of the 8000ish bitlines would be connected to nothing. Here’s the logic diagram for a simple multiplexer. We now have the means of accessing any memory cell in this massive array; however,
17:16
to understand the three basic operations, reading, writing, and refreshing let’s add two elements to our layout: A sense amplifier at the bottom of each bitline, and a read and write driver outside of the column multiplexer. Let’s look at reading from a group of memory cells. First the read command and 31-bit address are sent from the CPU to the DRAM. The first 5 bits select a specific bank. The next step is to turn off all the wordlines in that bank,
17:48
thereby isolating all the capacitors, and then precharge all 8000ish bitlines to .5 volts. Next the 16-bit row address turns on a row, and all the capacitors in that row are connected to their bitlines. If an individual capacitor holds a 1 and is charged to 1 volt, then some charge flows from the capacitor onto the .5-volt bitline, and the voltage on the bitline increases. The sense amplifier then detects this slight change or perturbation of voltage on the bitline,
18:22
amplifies the change, and pushes the voltage on the bitline all the way up to 1 volt. However, if a 0 is stored in the capacitor, charge flows from the bitline into the capacitor, and the .5-volt bitline decreases in voltage. The sense amplifier then sees this change, amplifies it and drives the bitline voltage down to 0 volts or ground. The sense amplifier is necessary because the capacitor is so small, and the bitline is rather long, and thus the
18:54
capacitor needs to have an additional component to sense and amplify whatever value is stored. Now, all 8000ish bitlines are driven to 1 volt or 0 volts corresponding to the stored charge in the capacitors of the activated row, and this row is now considered open. Next, the column select multiplexer uses the 10-bit column address to connect the corresponding 8 bitlines to the read driver which then sends these 8 values
19:24
and voltages over the 8 data wires to the CPU. Writing data to these memory cells is similar to reading, however with a few key differences. First the write command, address, and 8 bits to be written are sent to the DRAM chip. Next, just like before the bank is selected, the capacitors are isolated, and the bitlines are precharged to .5 volts. Then, using a 16-bit address, a single row is activated, the capacitors perturb the bitline, and the sense amplifiers sense this
20:01
and drive the bitlines to a 1 or 0 thus opening the row. Next the column address goes to the multiplexer, but, this time, because a write command was sent, the multiplexer connects the specific 8 bitlines to the write driver which contains the 8 bits that the CPU had sent along the data wires and requested to write. These write drivers are much stronger than the sense amplifier and thus they override whatever voltage was previously on the bitline, and drive each of
20:32
the 8 bitlines to 1 volt for a 1 to be written, or 0 volts for a 0. This new bitline voltage overrides the previously stored charges or values in each of the 8 capacitors in the open row, thereby writing 8 bits of data to the memory cells corresponding to the 31-bit address. Three quick notes. First, as a reminder, writing and reading happens concurrently with all the 4 chips in the shared memory channel, using the same 31-bit address and command wires,
21:03
but with different data wires for each chip. Second, with DDR5 for a binary 1 the voltage is actually 1.1 volts, for DDR4 it’s 1.2 volts, and prior generations had even higher voltages, with the bitline precharge voltages being half of these voltages. However, for DDR5, when writing or refreshing a higher voltage, around 1.4 volts is applied and stored in each capacitor for a binary 1 because charge leaks out over time. However, for simplicity, we’re
21:39
going to stick with 1 and 0. Third, the number of bank groups, banks, bitlines and wordlines varies widely between different generations and capacities but is always in powers of 2. Let’s move on and discuss the third operation which is refreshing the memory cells in a bank. As mentioned earlier, the transistors used to isolate the capacitors are incredibly small, and thus charges leak across the channel. The refresh operation is rather simple and is a
22:11
sequence of closing all the rows, precharging the bitlines to .5 volts, and opening a row. To refresh, just as before, the capacitors perturb the bitlines and then the sense amplifiers drive the bitlines and capacitors of the open row fully up to 1 volt or down to 0 volts depending on the stored value of the capacitor, thereby refilling the leaked charge. This process of row closing,
22:38
precharging, opening, and sense amplifying happens row after row, taking 50 nanoseconds for each row, until all 65 thousandish rows are refreshed taking a total of 3 milliseconds or so to complete. The refresh operation occurs once every 64 milliseconds for each bank, because that’s statistically below the worst-case time it takes for a memory cell to leak too much charge to make a stored 1 turn into a 0, thus resulting in a loss of data.
23:12
Let’s take a step back and consider the incredible amount of data that is moved through DRAM memory cells. These banks of memory cells handle up to 4 thousand 8 hundred million requests to read and write data every second while refreshing every memory cell in each bank row by row around 16 times a second. That’s a staggering amount of data movement and illustrates the true strength of computers. Yes, they do simple things like comparisons,
23:44
arithmetic, and moving data around, but at a rate of billions of times a second. Now, you might wonder why computers need to do so much data movement. Well, take this video game for example. You have obvious calculations like the movement of your character and the horse. But then there are individual grasses, trees, rocks, and animals whose positions and geometries are stored in DRAM. And then the environment such as the lighting
24:14
and shadows change the colors and textures of the environment in order to create a realistic world. Next, we’re going to explore breakthroughs and optimizations that allow DRAM to be incredibly fast. But, before we get into all those details, we would greatly appreciate it if you could take a second to hit that like button, subscribe if you haven’t already, and type up a quick comment below, as it helps get this video out to others. Also, we have a Patreon
24:45
and would appreciate any support. This is our longest and most detailed video by far, and we’re planning more videos that get into the inner details of how computers work. We can’t do it without your help, so thank you for watching and doing these three quick things. It helps a ton.
25:07
The first complex topic which we’ll explore is why there are 32 banks, as well as what the parameters on the packaging of DRAM are. After that, we’ll explore burst buffers, sub-arrays, and folded DRAM architecture and what’s inside the sense amplifier. Let’s take a look at the banks. As mentioned earlier opening a single row within a bank requires all these steps and this process takes time. However, if a row were already open, we could read or write to any section of
25:39
8 memory cells using only the 10-bit column address and the column select multiplexer. When the CPU sends a read or write command to a row that’s already open, it’s called a row hit or page hit, and this can happen over and over. With a row hit, we skip all the steps required to open a row, and just use the 10-bit column address to multiplex a different set of 8 columns or bitlines, connecting them to the read or write driver, thereby saving
26:09
a considerable amount of time. A row miss is when the next address is for a different row, which requires the DRAM to close and isolate the currently open row, and then open the new row. On a package of DRAM there are typically 4 numbers specifying timing parameters regarding row hits, precharging, and row misses. The first number refers to the time it takes between sending an address with a row open, thus a row hit, to receiving the data stored in those columns.
26:38
The next number is the time it takes to open a row if all the lines are isolated and the bitlines are precharged. Then the next number is the time it takes to precharge the bitlines before opening a row, and the last number is the time it takes between a row activation and the following precharge. Note that these numbers are measured in clock cycles. Row hits are also the reason why the address is sent in two sections, first the bank selection and
27:07
row address called RAS and then the column address called CAS. If the first part, the bank selection and row address, matches a currently open row, then it’s a row hit, and all the DRAM needs is the column address and the new command, and then the multiplexer simply moves around the open row. Because of the time saving in accessing an open row, the CPU memory controller, programs, and compilers are optimized for increasing the number of subsequent row hits. The opposite,
27:38
called thrashing, is when a program jumps around from one row to a different row over and over, and is obviously incredibly inefficient both in terms of energy and time. Additionally, DDR5 DRAM has 32 banks for this reason. Each bank’s rows, columns, sense amplifiers and row decoders operate independently of one another, and thus multiple rows from different banks can be open all at the same time, increasing the likelihood of a row hit,
28:09
and reducing the average time it takes for the CPU to access data. Furthermore, by having multiple bank groups, the CPU can refresh one bank in each bank group at a time while using the other three, thus reducing the impact of refreshing. A question you may have had earlier is why are banks significantly taller than they are wide? Well, by combining all the banks together one next to the other you can think of this chip as actually being 65 thousand rows tall by 262
28:44
thousand columns wide. And, by adding 31 equally spaced divisions between the columns, thus creating banks, we allow for much more flexibility and efficiency in reading, writing and refreshing. Also, note that on the DRAM packaging are its capacity in Gigabytes, the number of millions of data transfers per second, which is two times the clock frequency, and the peak data transfer rate in Megabytes per second. The next design optimization we’ll explore
29:16
is the burst buffer and burst length. Let’s add a 128-bit read and write temporary storage location, called a burst buffer to our functional diagram. Instead of 8 wires coming out of the multiplexer, we’re going to have 128 wires that connect to these 128-bit buffer locations. Next the 10-bit column address is broken into two parts, 6 bits are used for the multiplexer, and 4 bits are for the burst buffer. Let’s explore a reading command. With
29:49
our burst buffer in place, 128 memory cells and bitlines are connected to the burst buffer using the 6 column bits, thereby temporarily loading, or caching 128 values into the burst buffer. Using the 4 bits for the buffer, 8 quickly accessed data locations in the burst buffer are connected to the read drivers and the data is sent to the CPU. By cycling through these 4 bits, all 16 sets of 8 bits are read out, and thus the burst length is 16. After that a new set of 128
30:25
bitlines and values are connected and loaded into the burst buffer. There’s also a write burst buffer which operates in a similar way. The benefit of this design is that 16 sets of 8 bits per microchip, totaling 1024 bits, can be accessed and read or written extremely quickly, as long as the data is all next to one another, but at the same time we still have the granularity and ability to access any set of 8 bits if our data requests jump around.
30:56
The next design optimization is that this bank of 65536 rows by 8192 columns is rather massive, and results in extremely long wordlines and bitlines, especially when compared to the size of each trench capacitor memory cell. Therefore, the massive array is broken up into smaller blocks 1,024 by 1,024, with intermediate sense amplifiers below each subarray, and subdividing wordlines and using a hierarchical row decoding scheme. By subdividing the bitlines,
31:34
the distance and amount of wire that each tiny capacitor is connected to as it perturbs the bitline to the sense amplifier is reduced, and thus the capacitor doesn’t have to be as big. By subdividing the wordlines the capacitive load from eight thousandish transistor gates and channels is decreased, and thus the time it takes to turn on all the access transistors in a row is decreased. The final topic we’re going to talk about is the most complicated. Remember how we had
32:05
a sense amplifier connected to the bottom of each bitline? Well, this optimization has two bitlines per column going to each sense amplifier and alternating rows of memory cells connected to the left and right bitlines, thus doubling the number of bitlines. When one row is active, half of the bitlines are active while the other half are passive and vice versa when the next row is active. Moving down to see inside the sense amplifier we find a cross-coupled inverter. How
32:34
does this work? Well, when the active bitline is a 1, the passive bitline will be driven by this cross-coupled inverter to the opposite value of 0, and when the active is a 0, the passive becomes a 1. Note that the inverted passive bitline isn’t connected to any memory cells, and thus it doesn’t mess up any stored data. The cross-coupled inverter makes it such that these two bitlines are always going to be opposite one another, and they’re called a differential
33:03
pair. There are three benefits to this design. First, during the precharge step, we want to bring all the bitlines to .5 volts and, by having a differential pair of active and passive bitlines, the easiest solution is to disconnect the cross coupled inverters and open a channel between the two using a transistor. The charge easily flows from the 1 bitline to the 0, and they both average out and settle at .5 volts. The other two benefits are noise immunity,
33:34
and a reduction in parasitic capacitance of the bitline. These benefits are related to that fact that by creating two oppositely charged electric wires with electric fields going from one to the other we reduce the amount of electric fields emitted in stray directions and relatedly increase the ability of the sense amplifier to amplify one bitline to 1 volt and the other to 0 volts. One final note is that when discussing DRAM, one major topic is the timing of addresses,
34:04
command signals and data, and the related acronyms DDR or double data rate, and SDRAM, or Synchronous DRAM. These topics were omitted from this video because it would have taken an additional 15 minutes to properly explore. That’s
34:24
pretty much it for the DRAM, and we are grateful you made it this far into the video. We believe the future will require a strong emphasis on engineering education and we’re thankful to all our Patreon and YouTube Membership Sponsors for supporting this dream. If you want to support us on YouTube Memberships, or Patreon, you can find the links in the description. A huge thanks goes to the Nathan, Peter, and Jacob who are doctoral students at the Florida
34:53
Institute for Cybersecurity Research for helping to research and review this video’s content! They do foundational research on finding the weak points in device security and whether hardware is compromised. If you want to learn more about the FICS graduate program or their work, check out the website using the link in the description. This is Branch Education, and we create 3D animations that dive deep into the technology that drives our modern world. Watch another Branch
35:24
video by clicking one of these cards or click here to subscribe. Thanks for watching to the end!
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat