Skip to main content
Loading Explore guides…
Ray Tracing · Acedra
Home
Explore
Computer Science
Ray Tracing
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
55 segments
Copy
Hide video
0:00
Every new TV show and movie that uses computer-generated images and special effects relies on Ray Tracing. For example, in order to build an interstellar battle, set in a galaxy far, far away, 3D artists model and texture the spaceships, position them around the scene with lights, a background, and a camera, and then render the scene. Rendering is a computational process that simulates how rays of light bounce off of and
0:31
illuminate each of the models, thus transforming a scene full of simple 3D models into a realistic environment. There are many different ray tracing algorithms used to render scenes, but the current industry standard in TV shows and movies is called path tracing. This algorithm requires an unimaginable number of calculations. For example, if you had the entire population of the world working together and performing 1 calculation every second,
1:02
it would take 12 days of nonstop problem solving to turn this scene into this image. Due to these incredible computational requirements, path tracing was considered impossible for anything but super computers for decades. In fact, this algorithm for simulating light was first conceptualized in 1986, however it took 30 years before movies like Zootopia, Moana, Finding Dory and Coco could be rendered using path tracing and even then, rendering these movies required
1:38
a server farm of 1000s of computers and multiple months to complete. So, why does path tracing require quadrillions of calculations? And how does Ray Tracing work? Well, in this video, we’ll answer these two questions, and in the process, you’ll get a better understanding of how Computer Generated Images or CGI and special effects are created for TV and movies. After that we’ll open up this GPU and see how its architecture is specifically designed to execute ray tracing,
2:14
enabling it to render this scene in only a few minutes. And finally, we’ll investigate how Video games like Cyberpunk or the Unreal Engine Lumen Renderer use Ray Tracing. So, let’s dive right in.
2:36
This video is sponsored by Brilliant.org Let’s first see how Path Tracing works and how this dragon and kingdom are created and turned into a setting for a fantasy show. To make the scene, an artist first spends a few months modeling everything, the islands, the castles, the houses, the trees, and of course, the dragon. Although these models may have some smooth curves or squares and polygons, they’re actually all broken down into small triangles. In short,
3:10
GPUs almost exclusively work with 3D scenes made of triangles, and this scene is built from 3.2 million triangles. After a model is built, the 3D artist assigns a texture to it which defines both the color, as well as material attributes, such as whether the surface is rough, smooth, metallic, glass, water-like, or composed of a wide range of other materials. Next, the completed models are properly positioned around the scene and the artist adds
3:44
lights such as the sky and the sun and adjusts their intensity and direction to simulate the time of day. Finally, a virtual camera is added and the scene is rendered and brought to life. As mentioned earlier, path tracing simulates how light interacts with and bounces off every surface in the scene, thereby producing realistic effects such as smooth shadows across the buildings or the way light interacts with the water and produces bright highlights
4:15
in some areas and water covered sand in others. In the real world, light rays start at the sun, and when they hit a surface such as this red roof, some light is absorbed while the red light is reflected, thus tinting the light based on the color of the object. These now tinted light rays bounce off the surface and make their way to the camera and produce a 2D image. With this scene, a near infinite number of light rays are produced by the sun
4:47
and sky and only a small fraction of them actually reach the camera. Calculating an infinite number of light rays is impossible and only the light rays that reach the camera are useful, and therefore with path tracing we don’t send rays out from the sky or light source, but rather we send out rays from a virtual camera and into the scene. We then determine which objects the rays hit and calculate how those objects are illuminated by the light sources.
5:19
With computer-generated images or CGI, the 2D image is represented by a view plane in front of the virtual camera. This view plane has the same pixel count as the final image, so a 4K image has 8.3 million pixels. Furthermore, by animating the camera around or changing its field of view, the view plane will correspondingly change. Let’s transition to an indoor scene such as this barbershop, which contains 8 million triangles and is actually more complicated than
5:55
the island kingdom. In order to create this image on the view plane, a total of 8.3 billion rays, which are a thousand rays per pixel, are sent out from the virtual camera through the view plane and into the scene. Ray Tracing is a massively parallel operation because each pixel is independent from all other pixels. This means that the thousand rays from one pixel can be calculated at the same time as the rays from the next pixel over and so on. Once a single pixel’s rays finish
6:29
flying around the scene, the results are combined with the other rays and pixels to form a single image. If we were to show billions of rays, the scene would quickly become inundated with lines, so let’s simplify it down to a single ray running through one pixel of the viewing plane. This ray starts at the camera, travels through a random point in the pixel and into the scene. It flies straight and eventually hits a triangle, and once it does, that object’s color becomes
7:01
associated with that ray and pixel. For example, when the ray hits this chair, then the pixel becomes red. The other nearby rays running through random places in the same pixel will hit pretty close to this ray and have their colors averaged together. These rays are called primary rays and they answer the question of what triangle and object do the rays first hit and what basic color
7:27
should be in that specific pixel. Another example is that these rays running through this pixel hit the blue stripe on the barbershop pole turning the pixel blue. The other billions of rays do the same thing resulting in a single image with the proper 3D perspective from the virtual camera. This image is fairly flat colored because each pixel just has the simple color of the object the rays hit.
7:55
So the next question is: how is the location where the primary ray hits illuminated by the light sources and how bright or dark should the pixel’s color be shaded. For example, when you look at the blue stripe of the barbershop pole, the entire stripe is just blue, but in the rendered image, there’s a gradient from bright to dark across a number of pixels depending on how the triangles are facing the lights and the window. Specifically, the dark blue backside doesn’t
8:28
face any of the light sources and therefore its illumination comes only from light bouncing off the nearby walls. Furthermore, when the lighting conditions change and more light enters the scene, the entire barbershop pole brightens up. This accurate lighting applies to all the objects in the scene and is what transforms the scene and makes it look realistic. In order to accurately determine the brightness of these blue pixels, ray tracing first needs to determine how the surface
9:00
is illuminated directly by the light sources, which is called direct illumination, and second, how the surface is illuminated by light bouncing off other objects, which is called indirect illumination. Combining direct and indirect illumination is called global illumination. In order to calculate direct illumination, we start at the intersection point where the primary ray hits the triangle in the barbershop pole and then we generate additional rays called shadow
9:31
rays and send them in the direction of each light source such as the light bulbs or the sun outside the window. If there are no objects between the intersection point and a light source, then that means that this point on the blue stripe is directly illuminated by that light source. For each light source that directly illuminates this point, we factor in the light source’s brightness, size, color, distance, and the direction of the surface that the triangle inside the blue stripe
10:01
is facing. All these factors are multiplied by the Red, Blue, and Green or RGB values of the blue stripe, which in turn changes the shading or brightness of the pixel that the primary ray went through. Let’s brighten the room again, and you can see the RGB values increase for this pixel. Now let’s dim the room once more and look at a different pixel whose primary ray hits the backside of the barbershop pole. A similar set of shadow rays are sent out from this intersection
10:34
point to each light source, but each of these rays is blocked by other triangles in the pole, and thus this point doesn’t receive any direct illumination from any of the light sources, leaving the pixel dark. These rays are called shadow rays because they determine whether a location is directly illuminated by a light source or whether it’s in a shadow. You might think that this backside should be entirely black because it’s in the shadows and
11:03
none of the light rays from the light sources can reach it. However, this backside still has color because it’s illuminated by light bouncing off the walls. This light is called indirect illumination, and in order to calculate it, we take the intersection point from the primary ray and generate a secondary ray that bounces off it. This secondary ray then hits a new surface such as this point on the wall. From this secondary point we send out a new set of shadow rays to each light
11:36
source to see whether the point on the wall is in shadows or whether it’s directly illuminated. The results from these new shadow rays and the attributes of the corresponding light sources are combined with the color of the wall’s surface, essentially turning this point on the wall into a light source that illuminates the backside of the barbershop pole. Sometimes this point is still in shadows, so we create an additional secondary ray from the point on the wall and
12:07
send it in a new direction and see what it hits. Then we calculate how that third point is directly illuminated using yet another set of shadow rays thereby turning this third point into a light source that illuminates the previous point. This secondary ray bouncing happens multiple times, and each time we send shadow rays to the light sources and check how that point is illuminated. The purpose of bouncing the secondary rays around and sending out shadow rays at each
12:38
point is to find different paths where light bounces off different surfaces and indirectly illuminates the original point where the primary ray hits. Furthermore, by sending a thousand rays through random points in a single pixel, and by having thousands of secondary rays bounce in different directions, we get an accurate approximation for indirect illumination or how this pixel is illuminated by light bouncing off the other objects.
13:09
It's called path tracing because by using these primary rays, secondary rays and shadow rays, we’re finding billions of paths from the camera through different points in the scene and to the light sources. One additional benefit of indirect illumination and the use of secondary rays is that color can bounce from one object to another. For example, when we place a red balloon next to the wall and brighten the scene, some secondary light rays are tinted red by the balloon, and this
13:42
reddish color can be seen on the wall itself. An important detail is that the direction the secondary rays bounce off the surface depends on the material and texture properties assigned to the object. For example, here is a set of spheres that are all gray, but have different roughness values that drastically change their look. Essentially, for a perfectly smooth surface with no roughness, the object becomes a mirror because every one of the
14:12
secondary rays will bounce off in the same perfect reflection direction, and whatever the secondary rays hit will combine together and become visible in the mirror-like surface. However, when a material has a roughness set to 100%, then the secondary rays will bounce in entirely random directions resulting in a flat gray surface. Furthermore, if an object is assigned a glass material, then additional refraction rays that pass through the glass are generated,
14:46
and the color and brightness of the pixels in the glass will depend mostly on the direction of the refraction rays and what those rays hit. Here’s an interesting scene of some glass and mirror objects that truly show the power of path tracing, and you can see multiple mirror bounces in some of the objects and proper refraction in the glass. Note that for this barbershop scene a thousand
15:11
rays per pixel and four secondary bounces are the render settings we chose during scene setup. Other scenes use different numbers of rays per pixel, secondary bounces, and light sources. When we multiply these values together with the number of pixels in an image we get the total number of rays required to generate a single image. Furthermore, animations typically have 24 frames a second, so a 20-minute animation requires over a quadrillion rays, and that’s why path tracing
15:46
was considered computationally impossible for TV shows and movies for decades. The other key problem was figuring out which one triangle out of 8 million each of the rays hits first. So let’s see how these problems are solved and we’ll start by transitioning to a new scene and see how ray-triangle intersections are calculated. Let’s simplify the scene down to one ray and two triangles and find which one the ray hits. We start by extending the planes that the triangles
16:21
are on and then, using the equations of the planes and the ray, we calculate the point at which they intersect. Now that we have a set of intersection points on separate planes, we find whether the point is inside each corresponding triangle. If it is, then that means the ray hits the triangle, and if it isn’t that means it misses the triangle. These steps are relatively simple, and with 10 triangles, we can do this over and over, once for each triangle. If multiple
16:55
triangles are hit we do a distance calculation to find the closest one. However, when a scene has millions of triangles, finding which one triangle a single ray hits first becomes incredibly repetitive and computationally problematic. We solve this by using what’s called a bounding volume hierarchy or BVH. Essentially, we take triangles in the scene and, using their 3D coordinates, we divide them into two separate boxes called bounding volumes. Each of these
17:30
boxes contains half of all the triangles in the scene. Then we take these 2 boxes with their 1.5 million triangles and divide them again into boxes with 750,000 triangles. We keep on dividing the triangles into more and more progressively smaller pairs of boxes for a total of 19 divides. In the end we’ve separated 3 million triangles into a hierarchy of 19 divisions of boxes with a total of 525 thousand very small boxes at the bottom, each with around 6 triangles inside.
18:10
The key is that all of these boxes have their sides aligned with the coordinate axes, which makes a far easier calculation. For example e, if we have a ray and two axes aligned boxes, finding whether it hits box A or box B is just a matter of finding the intercept with the plane of Y equals six, and then seeing whether the intercept coordinates fall between box A’s bounds or between Box B’s bounds. Then we do the same thing inside Box B but using the axes aligned
18:45
coordinates of the two smaller boxes inside of it. For a scene of 3 million triangles, these 19 box divide branches form a binary tree or hierarchy, hence the name bounding volume hierarchy. At each branch we perform a simple ray-box intersection calculation to see which box the ray hits first, and then the ray travels to the next branch. At the very bottom, once a ray finishes traveling through all the bounding volume branches, which is called BVH traversal, we end up
19:22
with a small box of only 6 triangles. We then do the ray-triangle intersection calculation that we mentioned earlier with just these 6 triangles. As a result, BVH trees and traversal reduce tens of millions of calculations down to a handful of simple ray box intersections followed by 6 ray triangle intersections. Using BVHs helps to solve which triangle a ray will hit first but doesn’t fix the fact that a single frame of animation requires over
19:57
a hundred billion rays. The solution is in the incredibly powerful GPUs we now have. When we open up this GPU, we find a rather large microchip that has 10496 CUDA or shading cores and 82 Ray Tracing or RT cores. The CUDA cores perform basic arithmetic while the ray tracing cores are specially designed and optimized to execute Ray Tracing. Inside the RT cores are two sections, the BVH traversal section takes in all the coordinates of the boxes and the
20:35
direction of the ray and executes BVH traversal in nanoseconds. Then, the ray triangle intersection section uses the coordinates of the six or so triangles in the smallest bounding volume and quickly finds which triangle the ray hits first. The RT cores operate in parallel with one another and pipeline the operations so that a few billion rays can be handled every second, and a complex
21:03
scene like this one can be rendered in 4 minutes. Overall Path Tracing’s computationally impossible problems are solved by using bounding volume hierarchies along with improvements in GPU hardware. One crazy fact is that the most powerful supercomputer in the year 2000 was the ASCI White, which cost 110 million dollars and could perform 12.3 trillion operations a second. Compare this with the NVidia 3090 GPU which cost a few thousand dollars when it first came out in 2022
21:43
and the CUDA or shading cores perform 36 trillion operations a second. It’s mind-boggling how such an incredible amount of computing power can fit into a graphics card the size of a shoebox and how computer-generated images or CGI and special effects, which used to be only for high-budget films, can now be created on a desktop computer. Ray Tracing is a fusion of a variety of different disciplines from the physics of light, to trigonometry, vectors, and matrices, and then
22:18
also computer science, algorithms and hardware. Covering all these topics would require multiple hour-long videos which we don’t have time to do, but luckily Brilliant, the sponsor of this video, already has several free and easy to access courses that explore these topics. Brilliant is where you learn by doing, and is a website filled with thousands of fun and interactive modules, loaded with subjects ranging from the fundamentals of math to quantum mechanics to programming in
22:51
python to biology, and much more. When I learn new things on Brilliant, I like to think about Steve Jobs, and how he took a calligraphy class at college. Although at the time it had no practical application in his life, 10 years later when designing the Macintosh computer, he applied all the lessons from that calligraphy course to designing the typefaces and proportionally spaced fonts of the Mac. The key is that as you progress through Brilliant’s interactive
23:22
lessons and learn new things, you may not know how those lessons apply to your job or life, but there will be one or two courses that will click into place and change the trajectory of your career. However, if you don’t try out their courses, then you’ll never know. The other reason why Steve Jobs is applicable to ray tracing is because he was the CEO of Pixar from 1986 until 2006 and helped to design the computers that rendered some of its first movies. To be
23:54
a successful inventor like Steve Jobs, you need to be well versed in a wide range of disciplines. For the viewers of this channel, Brilliant is providing a free 30-day trial with access to all their thousands of lessons and is also offering 20% off an annual subscription. Just go to brilliant.org/brancheducation. The link is in the description below. We loved making this video because path tracing is an algorithm that we use daily
24:26
due to the fact that all our animations are created and rendered using a software called Blender which uses path tracing in its rendering engine. Specifically, here are all the scenes we used and some statistics that you can pause the video and look at. It takes a ton of work to create high quality educational videos. Researching this video, writing the script, and then animating the scenes has taken us over 800 hours,
24:56
so if you could take a quick second to like this video, subscribe to the channel, write a comment below and share it with someone who watches TV or movies it would help us a ton. Furthermore, we’d like to give a shout-out to the Blender Dev Team. Blender is an incredibly powerful, free-to-use modeling and animation software. Each of these scenes was made by an incredible artist and you can download them for free from the Blender website.
25:26
Finally, one question you may have is: how is ray tracing is used in video games. There are many different methods, so we’ll cover just a few of them. The first one is similar to path tracing but with some shortcuts. For a given environment in a video game, a very low-resolution duplicate of all the models in the scene is created. Path tracing is then used to determine direct and indirect lighting for each of these low-resolution objects and the results
25:58
are saved into a light map on the low-resolution duplicate. Then the light map is applied to the high-resolution version of the objects in the scene, creating realistic indirect lighting and shadows on the high-resolution objects. This method is pretty good at approximating indirect lighting and is one of the ray tracing techniques used in Unreal Engine’s Lumen renderer. The second and completely different method for using ray tracing in video games is called
26:28
screen space ray tracing. It doesn’t use the scene’s geometries but rather uses the images and data generated from the video game graphics rendering pipeline where all the objects in the scene undergo 3D transformations to build a flat 2D image on the viewscreen. During the video game graphics process, additional data is created, such as a depth map that shows how far each object and the corresponding pixels are from the camera, as well as a normal map
27:00
that shows the direction each of the objects and pixels are facing. By combining the view screen, the depth map, and the normal map, we can generate an approximation for the X, Y, and Z values of the various objects in the scene, as well as determine what direction each pixel is facing. Now that we have a simplified scene, let’s say this lake is reflective, and we want to know what pixels should be shown in its reflection. To figure it out, we use ray tracing with this
27:32
simplified screen space 3D representation and bounce the rays off of the lake’s pixels using the normal map. These rays then continue through the simplified geometry and hit the trees behind it, thus producing a reflection of the trees on the lake. One problematic issue with screen space ray tracing is that it can only use the data that’s on the screen. As a result, when the camera moves, the trees move out of view, and thus the trees are removed from the
28:03
screen space data and it’s impossible to see them in the reflection. Additionally, screen space ray tracing doesn’t allow for reflections of objects behind the camera. This type of ray tracing along with other rendering algorithms are used in games like Cyberpunk. Additionally, if you’re curious as to how video game graphics work, we have a separate video that explores all the steps such as Vertex Shading,
28:31
Rasterization, and Fragment Shading. The video game graphics rendering pipeline is entirely different from Ray Tracing, so we recommend you check it out. And, that’s pretty much it for Ray Tracing. We’d like to give a shoutout to Cem Yuksel, a professor at the School of Computing at the University of Utah. On his YouTube channel, you can find his lecture series on computer graphics and interactive graphics, which were
28:59
both instrumental in the research for this video. This is Branch Education, and we create 3D animations that dive deeply into the technology that drives our modern world. Watch another Branch video by clicking one of these cards or click here to subscribe.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat